Imaging for navigation systems and methods

ABSTRACT

Techniques are disclosed for systems and methods to provide passage planning for a mobile structure. A passage planning system includes a logic device configured to communicate with a user interface associated with the mobile structure and at least one operational state sensor mounted to or within the mobile structure. The logic device determines an operational range map based, at least in part, on an operational state of the mobile structure and/or environmental conditions associated with the mobile structure. Such operational range map and other control signals may be displayed to a user and/or used to generate a planned route and/or adjust a steering actuator, a propulsion system thrust, and/or other operational systems of the mobile structure.

CROSS REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of and priority to U.S. ProvisionalPatent Application No. 62/901,140 filed Sep. 16, 2019 and entitled“IMAGING FOR NAVIGATION SYSTEMS AND METHODS,” which is incorporatedherein by reference in its entirety

TECHNICAL FIELD

One or more embodiments of the invention relate generally to passageplanning and more particularly, for example, to systems and methods forsubstantially automated and environmentally compensated passage planningand/or general navigation for mobile structures.

BACKGROUND

Some contemporary mobile structures, particularly vehicles withelectrical propulsion systems, have limited ranges and difficultiesassociated with recharging their propulsion systems in areas outsidetheir home berth. Range projection for such mobile structures can bedependent upon a wide array of different factors, including chargelevels, environmental conditions (e.g., wind, current, sea state),desired traversal speed, availability of solar charging current, backupgenerator capacity, and/or other factors. Planning a passage or routefor such mobile structures can be difficult, and inaccurate planning canresult in unsafe situations and can convert such mobile structure into anavigational hazard for others.

Moreover, conducting such passage or route or while under conditions ofgeneral navigation of a sailboat or other vehicle with a potentially atleast partially obscured forward or heading view can present a risk ofinjury or damage to the vehicle, particularly when the passage or routeis constrained due to diminishing range capability. Thus, there is aneed for a methodology that can accurately and reliably provide generaland/or forward view monitoring for such mobile structures both whenplanning a passage or route and while traversing such planned route.

SUMMARY

Techniques are disclosed for systems and methods to provide passageplanning and/or general navigation for a mobile structure. In accordancewith one or more embodiments, a passage planning system may include alogic device, a memory, one or more sensors, one or moreactuators/controllers, and modules to interface with users, sensors,actuators, and/or other modules of a mobile structure. The logic devicemay be configured to determine an operational range map based, at leastin part, on an operational state of the mobile structure and/orenvironmental conditions associated with the mobile structure. Suchoperational range map and other control signals may be displayed to auser and/or used to adjust a steering actuator, a propulsion systemthrust, and/or other operational systems of the mobile structure.

In various embodiments, a passage planning system may include a userinterface associated with a mobile structure and a logic deviceconfigured to communicate with the user interface and at least oneoperational state sensor mounted to or within the mobile structure. Theuser interface may include a display, and the logic device may beconfigured to determine an operational state of the mobile structurebased, at least in part, on operational state data provided by the atleast one operational state sensor; determine environmental conditionsassociated with the mobile structure; and determine an operational rangemap associated with the mobile structure based, at least in part, on theoperational state of the mobile structure and the environmentalconditions associated with the mobile structure.

In some embodiments, a method to provide passage planning for a mobilestructure may include determining an operational state of a mobilestructure based, at least in part, on operational state data provided byat least one operational state sensor mounted to or within the mobilestructure; determining environmental conditions associated with themobile structure; and determining an operational range map associatedwith the mobile structure based, at least in part, on the operationalstate of the mobile structure and the environmental conditionsassociated with the mobile structure.

In other embodiments, a passage planning system may include a logicdevice configured to communicate with a ranging sensor system mounted toa mobile structure. The logic device may be configured to aggregateranging sensor data provided by the ranging sensor system into a passagedatabase; determine a set of potential navigation hazard contacts based,at least in part, on the passage database; and determine a set of hazardpriorities corresponding to the set of potential navigation hazardcontacts based, at least in part, on the passage database.

In some embodiments, a method to provide passage planning for a mobilestructure may include aggregating ranging sensor data provided by aranging sensor system mounted to a mobile structure into a passagedatabase; determining a set of potential navigation hazard contactsbased, at least in part, on the passage database; and determining a setof hazard priorities corresponding to the set of potential navigationhazard contacts based, at least in part, on the passage database.

In another embodiment, an imaging system for a passage planning and/ornavigation system may include an imaging module disposed within a sealedenclosure and configured to capture image data along an optical axis ofthe imaging system according to a field of view (FOV) of the imagingmodule; a mechanical roll stabilizer coupled to the imaging module,disposed within the sealed enclosure, and configured to providemechanical roll stabilization of the imaging module with respect to theoptical axis of the imaging system; and a logic device configured tocommunicate with the imaging system. The logic device may be configuredto receive orientation data from an orientation sensor coupled to themobile structure; determine a boresight roll of the imaging modulebased, at least in part, on the received orientation data; and controlthe mechanical roll stabilizer to compensate for the determinedboresight roll.

In some embodiments, a method to provide imagery for passage planningfor a mobile structure may include receiving orientation data from anorientation sensor coupled to a mobile structure; determining aboresight roll of an imaging module of an imaging system coupled tomobile structure based, at least in part, on the received orientationdata, where an imaging module is disposed within a sealed enclosure andconfigured to capture image data along an optical axis of the imagingsystem according to a field of view (FOV) of the imaging module; andcontrolling a mechanical roll stabilizer of the imaging system coupledto the imaging module to compensate for the determined boresight roll,where the mechanical roll stabilizer is disposed within the sealedenclosure and is configured to provide mechanical roll stabilization ofthe imaging module with respect to the optical axis of the imagingsystem.

The scope of the invention is defined by the claims, which areincorporated into this section by reference. A more completeunderstanding of embodiments of the invention will be afforded to thoseskilled in the art, as well as a realization of additional advantagesthereof, by a consideration of the following detailed description of oneor more embodiments. Reference will be made to the appended sheets ofdrawings that will first be described briefly.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A illustrates a block diagram of a mobile structure including apassage planning system in accordance with an embodiment of thedisclosure.

FIG. 1B illustrates a diagram of a watercraft including a passageplanning system in accordance with an embodiment of the disclosure.

FIG. 1C illustrates a diagram of a steering sensor/actuator for apassage planning system in accordance with an embodiment of thedisclosure.

FIG. 2 illustrates a diagram of an electric propulsion system for amobile structure with a passage planning system in accordance with anembodiment of the disclosure.

FIG. 3 illustrates a diagram of an electric propulsion system for amobile structure with a passage planning system in accordance with anembodiment of the disclosure.

FIG. 4 illustrates a graph of runtime and corresponding achievablelinear range as a function of vessel speed for an electric propulsionsystem of a mobile structure with a passage planning system inaccordance with an embodiment of the disclosure.

FIG. 5 illustrates a polar comparator chart showing an estimated speedof a mobile structure with a passage planning system when providedmotive force by electric power, wind, or a combination of wind andelectrical power, in accordance with an embodiment of the disclosure.

FIG. 6 shows a display view including an operational range map for amobile structure with a passage planning system in accordance with anembodiment of the disclosure.

FIGS. 7-10 illustrate flow diagrams of control loops to provide passageplanning in accordance with embodiments of the disclosure.

FIG. 11 illustrates a flow diagram of a process to provide rangeestimation and/or facilitate passage planning for a mobile structurewith a passage planning system in accordance with an embodiment of thedisclosure.

FIGS. 12A-B illustrate diagrams of an imaging system for use with apassage planning system in accordance with an embodiment of thedisclosure.

FIG. 13 shows an augmented reality (AR) display view for a mobilestructure with a passage planning system in accordance with anembodiment of the disclosure.

FIGS. 14A-C show AR display views for a mobile structure with a passageplanning system in accordance with an embodiment of the disclosure.

FIG. 15 shows an image processing strategy for AR display views for amobile structure with a passage planning system in accordance with anembodiment of the disclosure.

FIG. 16 illustrates a flow diagram of a process to provide AR displayviews for a mobile structure with a passage planning system inaccordance with an embodiment of the disclosure.

FIGS. 17A-B illustrate diagrams of imaging systems for use with apassage planning system in accordance with an embodiment of thedisclosure.

FIG. 18 illustrates a diagram of a dual element imaging system for usewith a passage planning system in accordance with an embodiment of thedisclosure.

FIG. 19 illustrates a diagram of a dual element imaging system enclosurefor use with a passage planning system in accordance with an embodimentof the disclosure.

FIGS. 20-21 show image module assemblies for a dual element imagingsystem for use with a passage planning system in accordance with anembodiment of the disclosure.

FIGS. 22A-B show AR display views including imagery provided by a dualelement imaging system for a mobile structure with a passage planningsystem in accordance with an embodiment of the disclosure.

FIGS. 23A-B show AR display views for a mobile structure with a passageplanning system in accordance with an embodiment of the disclosure.

FIG. 24 illustrates a flow diagram of a process to provide rollstabilized AR display views for a mobile structure with a passageplanning system in accordance with an embodiment of the disclosure.

Embodiments of the invention and their advantages are best understood byreferring to the detailed description that follows. It should beappreciated that like reference numerals are used to identify likeelements illustrated in one or more of the figures.

DETAILED DESCRIPTION

In accordance with various embodiments of the present disclosure,passage planning systems and methods may provide techniques for accurateand reliable range estimation and/or optimization for mobile structures,for example, to help navigate to a desired or safe destination within asafety margin (e.g., of fuel or battery storage) associated with themobile structure and/or selected by a pilot of the mobile structure.When vessels have limited range (e.g., electric or electric assistedwatercraft), the achievable range may be dependent upon many differentand interrelated factors, such as battery storage, water current, wind,vessel speed, sea state, solar generating capability, gasolinegenerating capability, accessory power use, and/or other factors.Selecting a strategy to safely and reliably arrive at a destination canbe a complex problem, and embodiments of system 100 may be configured tohelp users/pilots with passage planning and navigation according to adesired operational use of resources, as described herein.

In particular, for electric or electric assisted watercraft, such as asailboat with backup electric propulsion, determining the safe range ofpossible travel can be very complex. For example, a sailboat travellingacross 4 knots of wind may be able to make 2 knots under sail poweralone, 6 knots under full 5 kW electric motor power, or 5 knots undersail by applying 750 W of electric motor power. In various embodiments,the electric motor can provide a ‘gearing’ effect where a small amountof electric power provides enough motive force to adjust the apparentwind to allow the sails to work more efficiently. For such assistedsailing or e-sailing system coupled with 10 kWh of storage, theachievable range can be transformed from approximately 12 Nm (e.g.,using electric motor power alone) to 66 Nm by leveraging both electricand sail power while the electric power is available.

Other factors affecting range are wind direction and speed, watercurrent, sea state, the effects of sea state on a particular vessel,desired residual battery charge, generating capacity such as solar(photovoltaic) panels or gas generator, accessory (e.g., cooler,navigation, ranging) usage, minimum speed limits and/or other traversalrequirements (e.g., through shipping lanes), and/or other factors. For apilot wanting to select a port of arrival and plan a reliable passage,there are typically too many factors to consider while attempting tonavigate through traffic to a docking position. Thus, embodimentsprovide a graphical chart overlay or map to provide visualization ofsuch operational range of a mobile structure that accounts for thesefactors and provides a reliable estimation of range, as describedherein. Moreover, embodiments can use such operation range maps both tohelp plan a passage to a destination and to autopilot the mobilestructure through such planned passage or route safely.

FIG. 1A illustrates a block diagram of a passage planning system 100 inaccordance with an embodiment of the disclosure. In various embodiments,system 100 may be adapted to provide STM sensor calibration for aparticular mobile structure 101. In some embodiments, system 100 may beadapted to measure an orientation, a position, and/or a velocity ofmobile structure 101. System 100 may then use these measurements toprovide STM sensor calibration, which may then be used to controloperation of mobile structure 101, such as controlling elements ofnavigation control system 190 (e.g., steering actuator 150, propulsionsystem 170, and/or optional thrust maneuver system 172) to steer ororient mobile structure 101 according to a desired heading ororientation, such as heading angle 107, for example.

In the embodiment shown in FIG. 1A, system 100 may be implemented toprovide STM sensor calibration for a particular type of mobile structure101, such as a drone, a watercraft, an aircraft, a robot, a vehicle,and/or other types of mobile structures. In one embodiment, system 100may include one or more of a sonar system 110, a user interface 120, acontroller 130, an orientation sensor 140, a speed sensor 142, agyroscope/accelerometer 144, a global navigation satellite system (GNSS)146, a perimeter ranging system 148, a steering sensor/actuator 150, apropulsion system 170, a thrust maneuver system 172, and one or moreother sensors and/or actuators used to sense and/or control a state ofmobile structure 101, such as other modules 180. In some embodiments,one or more of the elements of system 100 may be implemented in acombined housing or structure that can be coupled to mobile structure101 and/or held or carried by a user of mobile structure 101.

Directions 102, 103, and 104 describe one possible coordinate frame ofmobile structure 101 (e.g., for headings or orientations measured byorientation sensor 140 and/or angular velocities and accelerationsmeasured by gyroscope/accelerometer 144). As shown in FIG. 1A, direction102 illustrates a direction that may be substantially parallel to and/oraligned with a longitudinal axis of mobile structure 101, direction 103illustrates a direction that may be substantially parallel to and/oraligned with a lateral axis of mobile structure 101, and direction 104illustrates a direction that may be substantially parallel to and/oraligned with a vertical axis of mobile structure 101, as describedherein. For example, a roll component of motion of mobile structure 101may correspond to rotations around direction 102, a pitch component maycorrespond to rotations around direction 103, and a yaw component maycorrespond to rotations around direction 104.

Heading angle 107 may correspond to the angle between a projection of areference direction 106 (e.g., the local component of the Earth'smagnetic field) onto a horizontal plane (e.g., referenced to agravitationally defined “down” vector local to mobile structure 101) anda projection of direction 102 onto the same horizontal plane. In someembodiments, the projection of reference direction 106 onto a horizontalplane (e.g., referenced to a gravitationally defined “down” vector) maybe referred to as Magnetic North. In various embodiments, MagneticNorth, a “down” vector, and/or various other directions, positions,and/or fixed or relative reference frames may define an absolutecoordinate frame, for example, where directional measurements referencedto an absolute coordinate frame may be referred to as absolutedirectional measurements (e.g., an “absolute” orientation).

In some embodiments, directional measurements may initially bereferenced to a coordinate frame of a particular sensor (e.g., a sonartransducer assembly or module of sonar system 110) and be transformed(e.g., using parameters for one or more coordinate frametransformations) to be referenced to an absolute coordinate frame and/ora coordinate frame of mobile structure 101. In various embodiments, anabsolute coordinate frame may be defined and/or correspond to acoordinate frame with one or more undefined axes, such as a horizontalplane local to mobile structure 101 referenced to a local gravitationalvector but with an unreferenced and/or undefined yaw reference (e.g., noreference to Magnetic North).

Sonar system 110 may be implemented with one or more electrically and/ormechanically coupled controllers, transmitters, receivers, transceivers,signal processing logic devices, autonomous power systems, variouselectrical components, transducer elements of various shapes and sizes,multichannel transducers/transducer modules, transducer assemblies,assembly brackets, transom brackets, and/or various actuators adapted toadjust orientations of any of the components of sonar system 110, asdescribed herein. Sonar system 110 may be configured to emit one,multiple, or a series of acoustic beams, receive corresponding acousticreturns, and convert the acoustic returns into sonar data and/orimagery, such as bathymetric data, water depth, water temperature, watercolumn/volume debris, bottom profile, and/or other types of sonar data.Sonar system 110 may be configured to provide such data and/or imageryto user interface 120 for display to a user, for example, or tocontroller 130 for additional processing, as described herein.

For example, in various embodiments, sonar system 110 may be implementedand/or operated according to any one or combination of the systems andmethods described in U.S. Provisional Patent Application 62/005,838filed May 30, 2014 and entitled “MULTICHANNEL SONAR SYSTEMS ANDMETHODS”, U.S. Provisional Patent Application 61/943,170 filed Feb. 21,2014 and entitled “MODULAR SONAR TRANSDUCER ASSEMBLY SYSTEMS ANDMETHODS”, and/or U.S. Provisional Patent Application 62/087,189 filedDec. 3, 2014 and entitled “AUTONOMOUS SONAR SYSTEMS AND METHODS”, eachof which are hereby incorporated by reference in their entirety. Inother embodiments, sonar system 110 may be implemented according toother sonar system arrangements that can be used to detect objectswithin a water column and/or a floor of a body of water.

User interface 120 may be implemented as one or more of a display, atouch screen, a keyboard, a mouse, a joystick, a knob, a steering wheel,a ship's wheel or helm, a yoke, and/or any other device capable ofaccepting user input and/or providing feedback to a user. For example,in some embodiments, user interface 120 may be implemented and/oroperated according to any one or combination of the systems and methodsdescribed in U.S. Provisional Patent Application 62/069,961 filed Oct.29, 2014 and entitled “PILOT DISPLAY SYSTEMS AND METHODS”, which ishereby incorporated by reference in its entirety.

In various embodiments, user interface 120 may be adapted to provideuser input (e.g., as a type of signal and/or sensor information) toother devices of system 100, such as controller 130. User interface 120may also be implemented with one or more logic devices that may beadapted to execute instructions, such as software instructions,implementing any of the various processes and/or methods describedherein. For example, user interface 120 may be adapted to formcommunication links, transmit and/or receive communications (e.g.,sensor signals, control signals, sensor information, user input, and/orother information), determine various coordinate frames and/ororientations, determine parameters for one or more coordinate frametransformations, and/or perform coordinate frame transformations, forexample, or to perform various other processes and/or methods describedherein.

In some embodiments, user interface 120 may be adapted to accept userinput, for example, to form a communication link, to select a particularwireless networking protocol and/or parameters for a particular wirelessnetworking protocol and/or wireless link (e.g., a password, anencryption key, a MAC address, a device identification number, a deviceoperation profile, parameters for operation of a device, and/or otherparameters), to select a method of processing sensor signals todetermine sensor information, to adjust a position and/or orientation ofan articulated sensor, and/or to otherwise facilitate operation ofsystem 100 and devices within system 100. Once user interface 120accepts a user input, the user input may be transmitted to other devicesof system 100 over one or more communication links.

In one embodiment, user interface 120 may be adapted to receive a sensoror control signal (e.g., from orientation sensor 140 and/or steeringsensor/actuator 150) over communication links formed by one or moreassociated logic devices, for example, and display sensor and/or otherinformation corresponding to the received sensor or control signal to auser. In related embodiments, user interface 120 may be adapted toprocess sensor and/or control signals to determine sensor and/or otherinformation. For example, a sensor signal may include an orientation, anangular velocity, an acceleration, a speed, and/or a position of mobilestructure 101 and/or other elements of system 100. In such embodiments,user interface 120 may be adapted to process the sensor signals todetermine sensor information indicating an estimated and/or absoluteroll, pitch, and/or yaw (attitude and/or rate), and/or a position orseries of positions of mobile structure 101 and/or other elements ofsystem 100, for example, and display the sensor information as feedbackto a user.

In one embodiment, user interface 120 may be adapted to display a timeseries of various sensor information and/or other parameters as part ofor overlaid on a graph or map, which may be referenced to a positionand/or orientation of mobile structure 101 and/or other element ofsystem 100. For example, user interface 120 may be adapted to display atime series of positions, headings, and/or orientations of mobilestructure 101 and/or other elements of system 100 overlaid on ageographical map, which may include one or more graphs indicating acorresponding time series of actuator control signals, sensorinformation, and/or other sensor and/or control signals.

In some embodiments, user interface 120 may be adapted to accept userinput including a user-defined target heading, waypoint, route, and/ororientation for an element of system 100, for example, and to generatecontrol signals for navigation control system 190 to cause mobilestructure 101 to move according to the target heading, waypoint, route,track, and/or orientation. In other embodiments, user interface 120 maybe adapted to accept user input modifying a control loop parameter ofcontroller 130, for example, or selecting a responsiveness of controller130 in controlling a direction (e.g., through application of aparticular steering angle) of mobile structure 101.

For example, a responsiveness setting may include selections ofPerformance (e.g., fast response), Cruising (medium response), Economy(slow response), and Docking responsiveness, where the differentsettings are used to choose between a more pronounced and immediatesteering response (e.g., a faster control loop response) or reducedsteering actuator activity (e.g., a slower control loop response). Insome embodiments, a responsiveness setting may correspond to a maximumdesired lateral acceleration during a turn. In such embodiments, theresponsiveness setting may modify a gain, a deadband, a limit on anoutput, a bandwidth of a filter, and/or other control loop parameters ofcontroller 130, as described herein.

In further embodiments, user interface 120 may be adapted to accept userinput including a user-defined target attitude, orientation, and/orposition for an actuated device (e.g., sonar system 110) associated withmobile structure 101, for example, and to generate control signals foradjusting an orientation and/or position of the actuated deviceaccording to the target attitude, orientation, and/or position. Moregenerally, user interface 120 may be adapted to display sensorinformation to a user, for example, and/or to transmit sensorinformation and/or user input to other user interfaces, sensors, orcontrollers of system 100, for instance, for display and/or furtherprocessing.

Controller 130 may be implemented as any appropriate logic device (e.g.,processing device, microcontroller, processor, application specificintegrated circuit (ASIC), field programmable gate array (FPGA), memorystorage device, memory reader, or other device or combinations ofdevices) that may be adapted to execute, store, and/or receiveappropriate instructions, such as software instructions implementing acontrol loop for controlling various operations of navigation controlsystem 190, mobile structure 101, and/or other elements of system 100,for example. Such software instructions may also implement methods forprocessing sensor signals, determining sensor information, providinguser feedback (e.g., through user interface 120), querying devices foroperational parameters, selecting operational parameters for devices, orperforming any of the various operations described herein (e.g.,operations performed by logic devices of various devices of system 100).

In addition, a machine readable medium may be provided for storingnon-transitory instructions for loading into and execution by controller130. In these and other embodiments, controller 130 may be implementedwith other components where appropriate, such as volatile memory,non-volatile memory, one or more interfaces, and/or various analogand/or digital components for interfacing with devices of system 100.For example, controller 130 may be adapted to store sensor signals,sensor information, parameters for coordinate frame transformations,calibration parameters, sets of calibration points, and/or otheroperational parameters, over time, for example, and provide such storeddata to a user using user interface 120. In some embodiments, controller130 may be integrated with one or more user interfaces (e.g., userinterface 120) and/or may share a communication module or modules.

As noted herein, controller 130 may be adapted to execute one or morecontrol loops to model or provide device control, steering control(e.g., using navigation control system 190) and/or performing othervarious operations of mobile structure 101 and/or system 100. In someembodiments, a control loop may include processing sensor signals and/orsensor information in order to control one or more operations of mobilestructure 101 and/or system 100.

For example, controller 130 may be adapted to receive a measured heading107 of mobile structure 101 from orientation sensor 140, a measuredsteering rate (e.g., a measured yaw rate, in some embodiments) fromgyroscope/accelerometer 144, a measured speed from speed sensor 142, ameasured position or series of absolute and/or relative positions fromGNSS 146, a measured steering angle from steering sensor/actuator 150,perimeter sensor data from perimeter ranging system 148, and/or a userinput from user interface 120. In some embodiments, a user input mayinclude a target heading 106, for example, an absolute position and/orwaypoint (e.g., from which target heading 106 may be derived), and/orone or more other control loop parameters. In further embodiments,controller 130 may be adapted to determine a steering demand or othercontrol signal for navigation control system 190 based on one or more ofthe received sensor signals, including the user input, and provide thesteering demand/control signal to steering sensor/actuator 150 and/ornavigation control system 190.

In some embodiments, a control loop may include a nominal vehiclepredictor used to produce a feedback signal corresponding to an averageor nominal vehicle/mobile structure rather than one specific to mobilestructure 101. Such feedback signal may be used to adjust or correctcontrol signals, as described herein. In some embodiments, a controlloop may include one or more vehicle dynamics modules corresponding toactual vehicles, for example, that may be used to implement an adaptivealgorithm for training various control loop parameters, such asparameters for a nominal vehicle predictor, without necessitatingreal-time control of an actual mobile structure.

Orientation sensor 140 may be implemented as one or more of a compass,float, accelerometer, and/or other device capable of measuring anorientation of mobile structure 101 (e.g., magnitude and direction ofroll, pitch, and/or yaw, relative to one or more reference orientationssuch as gravity and/or Magnetic North) and providing such measurementsas sensor signals that may be communicated to various devices of system100. In some embodiments, orientation sensor 140 may be adapted toprovide heading measurements for mobile structure 101. In otherembodiments, orientation sensor 140 may be adapted to provide a pitch,pitch rate, roll, roll rate, yaw, and/or yaw rate for mobile structure101 (e.g., using a time series of orientation measurements). In suchembodiments, controller 130 may be configured to determine a compensatedyaw rate based on the provided sensor signals. In various embodiments, ayaw rate and/or compensated yaw rate may be approximately equal to asteering rate of mobile structure 101. Orientation sensor 140 may bepositioned and/or adapted to make orientation measurements in relationto a particular coordinate frame of mobile structure 101, for example.

Speed sensor 142 may be implemented as an electronic pitot tube, meteredgear or wheel, water speed sensor, wind speed sensor, a wind velocitysensor (e.g., direction and magnitude) and/or other device capable ofmeasuring or determining a linear speed of mobile structure 101 (e.g.,in a surrounding medium and/or aligned with a longitudinal axis ofmobile structure 101) and providing such measurements as sensor signalsthat may be communicated to various devices of system 100. In someembodiments, speed sensor 142 may be adapted to provide a velocity of asurrounding medium relative to sensor 142 and/or mobile structure 101.For example, speed sensor 142 may be configured to provide an absoluteor relative wind velocity or water velocity impacting mobile structure101. In various embodiments, system 100 may include multiple embodimentsof speed sensor 142, such as one wind velocity sensor and one watervelocity sensor. In various embodiments, speed sensor 142 may bereferred to as an STM sensor.

Gyroscope/accelerometer 144 may be implemented as one or more electronicsextants, semiconductor devices, integrated chips, accelerometersensors, accelerometer sensor systems, or other devices capable ofmeasuring angular velocities/accelerations and/or linear accelerations(e.g., direction and magnitude) of mobile structure 101 and providingsuch measurements as sensor signals that may be communicated to otherdevices of system 100 (e.g., user interface 120, controller 130). Insome embodiments, gyroscope/accelerometer 144 may be adapted todetermine pitch, pitch rate, roll, roll rate, yaw, yaw rate, compensatedyaw rate, an absolute speed, and/or a linear acceleration rate of mobilestructure 101. Thus, gyroscope/accelerometer 144 may be adapted toprovide a measured heading, a measured steering rate, and/or a measuredspeed for mobile structure 101. In some embodiments,gyroscope/accelerometer 144 may provide pitch rate, roll rate, yaw rate,and/or a linear acceleration of mobile structure 101 to controller 130and controller 130 may be adapted to determine a compensated yaw ratebased on the provided sensor signals. Gyroscope/accelerometer 144 may bepositioned and/or adapted to make such measurements in relation to aparticular coordinate frame of mobile structure 101, for example. Invarious embodiments, gyroscope/accelerometer 144 may be implemented in acommon housing and/or module to ensure a common reference frame or aknown transformation between reference frames.

GNSS 146 may be implemented as a global positioning satellite receiverand/or other device capable of determining an absolute and/or relativeposition of mobile structure 101 based on wireless signals received fromspace-born and/or terrestrial sources, for example, and capable ofproviding such measurements as sensor signals that may be communicatedto various devices of system 100. In some embodiments, GNSS 146 may beadapted to determine and/or estimate a velocity, speed, and/or yaw rateof mobile structure 101 (e.g., using a time series of positionmeasurements), such as an absolute velocity and/or a yaw component of anangular velocity of mobile structure 101. In various embodiments, one ormore logic devices of system 100 may be adapted to determine acalculated speed of mobile structure 101 and/or a computed yaw componentof the angular velocity from such sensor information. GNSS 146 may alsobe used to estimate a relative wind velocity or a water currentvelocity, for example, using a time series of position measurementswhile mobile structure is otherwise lacking powered navigation control.

Perimeter ranging system 148 may be adapted to detect navigation hazardswithin a monitoring perimeter of mobile structure 101 (e.g., within apreselected or predetermined range of a perimeter of mobile structure101) and measure ranges to the detected navigation hazards (e.g., theclosest approach distance between a perimeter of mobile structure 101and a detected navigation hazard) and/or relative velocities of thedetected navigation hazards. In some embodiments, perimeter rangingsystem 148 may be implemented by one or more ultrasonic sensor arraysdistributed along the perimeter of mobile structure 101, radar systems,short range radar systems (e.g., including radar arrays configured todetect and/or range objects between a few centimeters and 10s of metersfrom a perimeter of mobile structure 101), visible spectrum and/orinfrared/thermal imaging modules or cameras, stereo cameras, LIDARsystems, combinations of these, and/or other perimeter ranging systemsconfigured to provide relatively fast and accurate perimeter sensor data(e.g., so as to accommodate suddenly changing navigation conditions dueto external disturbances such as tide and wind loadings on mobilestructure 101).

Navigation hazards, as used herein, may include an approaching dock ortie down post, other vehicles, floating debris, mooring lines, swimmersor water life, and/or other navigation hazards large and/or solid enoughto damage mobile structure 101, for example, or that require their ownsafety perimeter due to regulation, safety, or other concerns. As such,in some embodiments, perimeter ranging system 148 and/or controller 130may be configured to differentiate types of navigation hazards and/orobjects or conditions that do not present a navigation hazard, such asseaweed, pollution slicks, relatively small floating debris (e.g.,depending on a relative speed of the floating debris), and/or othernon-hazardous but detectable objects.

Steering sensor/actuator 150 may be adapted to physically adjust aheading of mobile structure 101 according to one or more controlsignals, user inputs, and/or stabilized attitude estimates provided by alogic device of system 100, such as controller 130. Steeringsensor/actuator 150 may include one or more actuators and controlsurfaces (e.g., a rudder or other type of steering mechanism) of mobilestructure 101 and may be adapted to sense and/or physically adjust thecontrol surfaces to a variety of positive and/or negative steeringangles/positions.

For example, FIG. 1C illustrates a diagram of a steering sensor/actuatorfor a passage planning system in accordance with an embodiment of thedisclosure. As shown in FIG. 1C, rear portion 101C of mobile structure101 includes steering sensor/actuator 150 configured to sense a steeringangle of rudder 152 and/or to physically adjust rudder 152 to a varietyof positive and/or negative steering angles, such as a positive steeringangle α measured relative to a zero steering angle direction (e.g.,designated by a dashed line 134). In various embodiments, steeringsensor/actuator 150 may be implemented with a steering actuator anglelimit (e.g., the positive limit is designated by an angle β, and adashed line 136 in FIG. 1C), and/or a steering actuator rate limit “R”.

As described herein, a steering actuator rate limit may be a limit ofhow quickly steering sensor/actuator 150 can change a steering angle ofa steering mechanism (e.g., rudder 132), and, in some embodiments, suchsteering actuator rate limit may vary depending on a speed of mobilestructure 101 along heading 104 (e.g., a speed of a ship relative tosurrounding water, or of a plane relative to a surrounding air mass). Infurther embodiments, a steering actuator rate limit may vary dependingon whether steering sensor/actuator 150 is turning with (e.g., anincreased steering actuator rate limit) or turning against (e.g., adecreased steering actuator rate limit) a prevailing counteractingforce, such as a prevailing current (e.g., a water and/or air current).A prevailing current may be determined from sensor signals provided byorientation sensor 140, gyroscope/accelerometer 142, speed sensor 144,and/or GNSS 146, for example.

In various embodiments, steering sensor/actuator 150 may be implementedas a number of separate sensors and/or actuators, for example, to senseand/or control one or more steering mechanisms substantiallysimultaneously, such as one or more rudders, elevators, and/orautomobile steering mechanisms, for example. In some embodiments,steering sensor/actuator 150 may include one or more sensors and/oractuators adapted to sense and/or adjust a propulsion force (e.g., apropeller speed and/or an engine rpm) of mobile structure 101, forexample, to effect a particular navigation maneuver (e.g., to meet aparticular steering demand within a particular period of time), forinstance, or to provide a safety measure (e.g., an engine cut-off and/orreduction in mobile structure speed).

In some embodiments, rudder 152 (e.g., a steering mechanism) may beimplemented as one or more control surfaces and/or conventional rudders,one or more directional propellers and/or vector thrusters (e.g.,directional water jets), a system of fixed propellers and/or thrustersthat can be powered at different levels and/or reversed to effect asteering rate of mobile structure 101, and/or other types or combinationof types of steering mechanisms appropriate for mobile structure 101. Inembodiments where rudder 152 is implemented, at least in part, as asystem of fixed propellers and/or thrusters, steering angle α mayrepresent an effective and/or expected steering angle based on, forexample, characteristics of mobile structure 101, the system of fixedpropellers and/or thrusters (e.g., their position on mobile structure101), and/or control signals provided to steering sensor/actuator 150.An effective and/or expected steering angle α may be determined bycontroller 130 according to a pre-determined algorithm, for example, orthrough use of an adaptive algorithm for training various control loopparameters characterizing the relationship of steering angle α to, forinstance, power levels provided to the system of fixed propellers and/orthrusters and/or control signals provided by controller 130, asdescribed herein.

Propulsion system 170 may be implemented as a propeller, turbine, orother thrust-based propulsion system, a mechanical wheeled and/ortracked propulsion system, a sail-based propulsion system, and/or othertypes of propulsion systems that can be used to provide motive force tomobile structure 101. In some embodiments, propulsion system 170 may benon-articulated, for example, such that the direction of motive forceand/or thrust generated by propulsion system 170 is fixed relative to acoordinate frame of mobile structure 101. Non-limiting examples ofnon-articulated propulsion systems include, for example, an inboardmotor for a watercraft with a fixed thrust vector, for example, or afixed aircraft propeller or turbine. In other embodiments, propulsionsystem 170 may be articulated, for example, and/or may be coupled toand/or integrated with steering sensor/actuator 150, such that thedirection of generated motive force and/or thrust is variable relativeto a coordinate frame of mobile structure 101. Non-limiting examples ofarticulated propulsion systems include, for example, an outboard motorfor a watercraft, an inboard motor for a watercraft with a variablethrust vector/port (e.g., used to steer the watercraft), a sail, or anaircraft propeller or turbine with a variable thrust vector, forexample. As such, in some embodiments, propulsion system 170 may beintegrated with steering sensor/actuator 150.

Optional thrust maneuver system 172 may be adapted to physically adjusta position, orientation, and/or linear and/or angular velocity of mobilestructure 101 according to one or more control signals and/or userinputs provided by a logic device of system 100, such as controller 130.Thrust maneuver system 172 may be implemented as one or more directionalpropellers and/or vector thrusters (e.g., directional water jets),and/or a system of fixed propellers and/or thrusters coupled to mobilestructure 101 that can be powered at different levels and/or reversed tomaneuver mobile structure 101 according to a desired linear and/orangular velocity.

Other modules 180 may include other and/or additional sensors,actuators, communications modules/nodes, and/or user interface devicesused to provide additional environmental information of mobile structure101, for example. In some embodiments, other modules 180 may include ahumidity sensor, a wind and/or water temperature sensor, a barometer, aradar system, a visible spectrum camera, an infrared camera, and/orother environmental sensors providing measurements and/or other sensorsignals that can be displayed to a user and/or used by other devices ofsystem 100 (e.g., controller 130) to provide operational control ofmobile structure 101 and/or system 100 that compensates forenvironmental conditions, such as wind speed and/or direction, swellspeed, amplitude, and/or direction, and/or an object in a path of mobilestructure 101, for example. In some embodiments, other modules 180 mayinclude one or more actuated and/or articulated devices (e.g.,spotlights, visible and/or IR cameras, radars, sonars, and/or otheractuated devices) coupled to mobile structure 101, where each actuateddevice includes one or more actuators adapted to adjust an orientationof the device, relative to mobile structure 101, in response to one ormore control signals (e.g., provided by controller 130).

In general, each of the elements of system 100 may be implemented withany appropriate logic device (e.g., processing device, microcontroller,processor, application specific integrated circuit (ASIC), fieldprogrammable gate array (FPGA), memory storage device, memory reader, orother device or combinations of devices) that may be adapted to execute,store, and/or receive appropriate instructions, such as softwareinstructions implementing any of the methods described herein, forexample, including for transmitting and/or receiving communications,such as sensor signals, sensor information, and/or control signals,between one or more devices of system 100. In various embodiments, suchmethod may include instructions for forming one or more communicationlinks between various devices of system 100.

In addition, one or more machine readable mediums may be provided forstoring non-transitory instructions for loading into and execution byany logic device implemented with one or more of the devices of system100. In these and other embodiments, the logic devices may beimplemented with other components where appropriate, such as volatilememory, non-volatile memory, and/or one or more interfaces (e.g.,inter-integrated circuit (I2C) interfaces, mobile industry processorinterfaces (MIPI), joint test action group (JTAG) interfaces (e.g., IEEE1149.1 standard test access port and boundary-scan architecture),controller area network (CAN) bus interfaces, and/or other interfaces,such as an interface for one or more antennas, or an interface for aparticular type of sensor).

Each of the elements of system 100 may be implemented with one or moreamplifiers, modulators, phase adjusters, beamforming components, digitalto analog converters (DACs), analog to digital converters (ADCs),various interfaces, antennas, transducers, and/or other analog and/ordigital components enabling each of the devices of system 100 totransmit and/or receive signals, for example, in order to facilitatewired and/or wireless communications between one or more devices ofsystem 100. Such components may be integrated with a correspondingelement of system 100, for example. In some embodiments, the same orsimilar components may be used to perform one or more sensormeasurements, as described herein.

Sensor signals, control signals, and other signals may be communicatedamong elements of system 100 using a variety of wired and/or wirelesscommunication techniques, including voltage signaling, Ethernet, WiFi,Bluetooth, Zigbee, Xbee, Micronet, CAN bus, or other medium and/or shortrange wired and/or wireless networking protocols and/or implementations,for example. In such embodiments, each element of system 100 may includeone or more modules supporting wired, wireless, and/or a combination ofwired and wireless communication techniques.

In some embodiments, various elements or portions of elements of system100 may be integrated with each other, for example, or may be integratedonto a single printed circuit board (PCB) to reduce system complexity,manufacturing costs, power requirements, coordinate frame errors, and/ortiming errors between the various sensor measurements. For example,gyroscope/accelerometer 144 and controller 130 may be configured toshare one or more components, such as a memory, a logic device, acommunications module, and/or other components, and such sharing may actto reduce and/or substantially eliminate such timing errors whilereducing overall system complexity and/or cost.

Each element of system 100 may include one or more batteries,capacitors, or other electrical power storage devices, for example, andmay include one or more solar cell modules or other electrical powergenerating devices (e.g., a wind or water-powered turbine, or agenerator producing electrical power from motion of one or more elementsof system 100). In some embodiments, one or more of the devices may bepowered by a power source for mobile structure 101, using one or morepower leads. Such power leads may also be used to support one or morecommunication techniques between elements of system 100.

In various embodiments, a logic device of system 100 (e.g., oforientation sensor 140 and/or other elements of system 100) may beadapted to determine parameters (e.g., using signals from variousdevices of system 100) for transforming a coordinate frame of otherelements of system 100 to/from a coordinate frame of mobile structure101, at-rest and/or in-motion, and/or other coordinate frames, asdescribed herein. One or more logic devices of system 100 may be adaptedto use such parameters to transform a coordinate frame of the otherelements of system 100 to/from a coordinate frame of orientation sensor140 and/or mobile structure 101, for example. Furthermore, suchparameters may be used to determine and/or calculate one or moreadjustments to an orientation of an element of system 100 that would benecessary to physically align a coordinate frame of the element with acoordinate frame of orientation sensor 140 and/or mobile structure 101,for example, or an absolute coordinate frame and/or other desiredpositions and/or orientations. Adjustments determined from suchparameters may be used to selectively power adjustment servos/actuators(e.g., of various elements of system 100), for example, or may becommunicated to a user through user interface 120, as described herein.

FIG. 1B illustrates a diagram of system 100B in accordance with anembodiment of the disclosure. In the embodiment shown in FIG. 1B, system100B may be implemented to provide STM sensor calibration and/oradditional operational control of mobile structure 101, similar tosystem 100 of FIG. 1B. For example, system 100B may include integrateduser interface/controller 120/130, secondary user interface 120,perimeter ranging system 148 a and 148 b, steering sensor/actuator 150,sensor cluster 160 (e.g., orientation sensor 140,gyroscope/accelerometer 144, and/or GNSS 146), and various other sensorsand/or actuators. In the embodiment illustrated by FIG. 1B, mobilestructure 101 is implemented as a motorized boat including a hull 105 b,a deck 106 b, a transom 107 b, a mast/sensor mount 108 b, a rudder 152,an inboard motor 170, articulated thrust maneuver jet 172, an actuatedsonar system 110 coupled to transom 107 b, perimeter ranging system 148a (e.g., a camera system, radar system, and/or LIDAR system) coupled tomast/sensor mount 108 b, optionally through roll, pitch, and/or yawactuator 162, and perimeter ranging system 148 b (e.g., an ultrasonicsensor array and/or short range radar system)) coupled to hull 105 b ordeck 106 b substantially above a water line of mobile structure 101. Inother embodiments, hull 105 b, deck 106 b, mast/sensor mount 108 b,rudder 152, inboard motor 170, and various actuated devices maycorrespond to attributes of a passenger aircraft or other type ofvehicle, robot, or drone, for example, such as an undercarriage, apassenger compartment, an engine/engine compartment, a trunk, a roof, asteering mechanism, a headlight, a radar system, and/or other portionsof a vehicle.

As depicted in FIG. 1B, mobile structure 101 includes actuated sonarsystem 110, which in turn includes transducer assembly 112 coupled totransom 107 b of mobile structure 101 through assembly bracket/actuator116 and transom bracket/electrical conduit 114. In some embodiments,assembly bracket/actuator 116 may be implemented as a roll, pitch,and/or yaw actuator, for example, and may be adapted to adjust anorientation of transducer assembly 112 according to control signalsand/or an orientation (e.g., roll, pitch, and/or yaw) or position ofmobile structure 101 provided by user interface/controller 120/130.Similarly, actuator 162 may be adapted to adjust an orientation ofperimeter ranging system 148 according to control signals and/or anorientation or position of mobile structure 101 provided by userinterface/controller 120/130. For example, user interface/controller120/130 may be adapted to receive an orientation of transducer assembly112 and/or perimeter ranging system 148 (e.g., from sensors embeddedwithin the assembly or device), and to adjust an orientation of eitherto maintain sensing/illuminating a position and/or absolute direction inresponse to motion of mobile structure 101, using one or moreorientations and/or positions of mobile structure 101 and/or othersensor information derived by executing various methods describedherein.

In one embodiment, user interfaces 120 may be mounted to mobilestructure 101 substantially on deck 106 b and/or mast/sensor mount 108b. Such mounts may be fixed, for example, or may include gimbals andother leveling mechanisms/actuators so that a display of user interfaces120 stays substantially level with respect to a horizon and/or a “down”vector (e.g., to mimic typical user head motion/orientation). In anotherembodiment, at least one of user interfaces 120 may be located inproximity to mobile structure 101 and be mobile throughout a user level(e.g., deck 106 b) of mobile structure 101. For example, secondary userinterface 120 may be implemented with a lanyard and/or other type ofstrap and/or attachment device and be physically coupled to a user ofmobile structure 101 so as to be in proximity to mobile structure 101.In various embodiments, user interfaces 120 may be implemented with arelatively thin display that is integrated into a PCB of thecorresponding user interface in order to reduce size, weight, housingcomplexity, and/or manufacturing costs.

As shown in FIG. 1B, in some embodiments, speed sensor 142 may bemounted to a portion of mobile structure 101, such as to hull 105 b, andbe adapted to measure a relative water speed, such as a STM or STWsensor configured to measure the relative speed of mobile structure 101through a surrounding water medium, substantially along longitudinalaxis 102. In some embodiments, speed sensor 142 may be adapted toprovide a relatively thin profile to reduce and/or avoid water drag. Invarious embodiments, speed sensor 142 may be mounted to a portion ofmobile structure 101 that is substantially outside easy operationalaccessibility. Speed sensor 142 may include one or more batteries and/orother electrical power storage devices, for example, and may include oneor more water-powered turbines to generate electrical power. In otherembodiments, speed sensor 142 may be powered by a power source formobile structure 101, for example, using one or more power leadspenetrating hull 105 b. In alternative embodiments, speed sensor 142 maybe implemented as a wind velocity sensor, for example, and may bemounted to mast/sensor mount 108 b to have relatively clear access tolocal wind.

In the embodiment illustrated by FIG. 1B, mobile structure 101 includesdirection/longitudinal axis 102, direction/lateral axis 103, anddirection/vertical axis 104 meeting approximately at mast/sensor mount108 b (e.g., near a center of gravity of mobile structure 101). In oneembodiment, the various axes may define a coordinate frame of mobilestructure 101 and/or sensor cluster 160. Each sensor adapted to measurea direction (e.g., velocities, accelerations, headings, or other statesincluding a directional component) may be implemented with a mount,actuators, and/or servos that can be used to align a coordinate frame ofthe sensor with a coordinate frame of any element of system 100B and/ormobile structure 101. Each element of system 100B may be located atpositions different from those depicted in FIG. 1B. Each device ofsystem 100B may include one or more batteries or other electrical powerstorage devices, for example, and may include one or more solar cellmodules or other electrical power generating devices. In someembodiments, one or more of the devices may be powered by a power sourcefor mobile structure 101. As noted herein, each element of system 100Bmay be implemented with an antenna, a logic device, and/or other analogand/or digital components enabling that element to provide, receive, andprocess sensor signals and interface or communicate with one or moredevices of system 100B. Further, a logic device of that element may beadapted to perform any of the methods described herein.

FIG. 2 illustrates a diagram of an electric propulsion system 200 formobile structure 101 with elements of passage planning system 100 ofFIGS. 1A-B, which may be implemented as an embodiment of propulsionsystem 170 and/or other elements of navigation system 190 of FIG. 1A. Inparticular, as shown in FIG. 2, electric propulsion system 200 mayinclude a shore power coupling/converter 210 receiving external power209 from a shore or dock hookup and providing external charging power211 to main battery 220, which may be used to store and delivermechanical propulsion power 221 to electric drive motor 230 to providepropulsive force to mobile structure 101 (e.g., through a propeller orother thrust system, as described herein). Electric propulsion system200 may also include fuel tank 240 providing fuel according to fuelconsumption 241 to combustion (e.g., diesel, gasoline) generator 242,which may be configured to provide generator charging power 243 tocharge main battery 220 and/or power electric drive motor 230.

In some embodiments, electric propulsion system 200 may additionally oralternatively include solar panels 250 providing solar charging power251 to leveling or ballast battery bank 252, which may be configured toprovide or receive exchange power 253 to charge main battery 220 and/orhelp level charge across multiple cells within main battery 220. Asshown in FIG. 2, electric propulsion system 200 may also include anaccessory power sink 224, such as a cooler or fridge, for example, or anelectronic navigation system (e.g., including elements of system 100 ofFIG. 1A), that draws or consumes accessory power 222 from main battery220. Electric propulsion system 200 may include a separate accessorybattery 260 (e.g., used to start generator 242 and/or provide emergencypower if main battery 220 is exhausted or damaged) and/or acommunications module 270 (e.g., to communicate to other vessels orshore-based systems, including remote storage systems, as describedherein).

Each of the elements of electric propulsion system 200 may beimplemented with power (e.g., voltage and/or current) monitoring and/orswitching circuitry to allow controller 130 of system 100, for example,to monitor and/or control power delivery to elements of electricpropulsion system 200. Moreover, main battery 220, leveling battery bank252, and accessory battery 260 may be implemented with charge statemonitoring circuitry to allow controller 130 to monitor the variouscharge states of each battery. In various embodiments, controller 130may be configured to receive an electrical status of electric propulsionsystem 200, including power delivery and/or charge state of each of theelements of electric propulsion system 200, and to determine an electricpropulsion state of electric propulsion system 200 based on such status.Such electric propulsion state (e.g., an operational state of mobilestructure 101) may be used, along with other factors, to determine anachievable range of mobile structure 101, as described herein.

FIG. 3 illustrates a diagram of an electric propulsion system 300 formobile structure 101 with elements of passage planning system 100 ofFIGS. 1A-B, which may be implemented as an embodiment of propulsionsystem 170 and/or other elements of navigation system 190 of FIG. 1A. Inparticular, as shown in FIG. 3, electric propulsion system 300 mayinclude one or more sails (e.g., mainsail 370 and/or headsail/spinnaker371) coupled to mast 108 b and configured to provide motive force tomobile structure 101, solar panels 250 a implemented as part of a sailbag for mainsail 370 as shown, and solar panels 250 b coupled to transom107 b of mobile structure 101. Each group of solar panels 250 a and/or250 b may be used to provide solar charging power to help charge mainbattery 220. Moreover, the expected or predicted power delivery of eachgroup of solar panels 250 a and/or 250 b may be dependent on time ofday, the absolute orientation of mobile structure 101 (e.g., relative tothe sun), the relative orientation of mainsail 370 and/or headsail 371,the sea state (e.g., level of chop or rolling waves obscuring solarpanels 250 b), and/or other factors, as described herein. Therelationship of power delivery to any one or group of such factors maybe characterized at manufacture, for example, or may be learned orsupplemented with state measurements (e.g., power delivery measurementscorrelated with measured operational states of mobile structure 101,including environmental states) made while underway, which may beaccumulated over time to refine such relationship.

Electric propulsion provides limited range that can be highly dependenton a selected vessel speed. FIG. 4 illustrates a graph 400 of runtime410 and corresponding achievable linear range 420 as a function ofvessel speed (e.g., assuming no water current effect and no assistancefrom wind) for electric propulsion system 200 and mobile structure 101of FIG. 2. As can be seen from FIG. 4, as the desired or selected speedis decreased, the corresponding range generally increases, due tovarious factors including hull shape, hull drag, vessel weight, andelectromechanical conversion efficiency, as described herein.

Motor-sailing (e.g., including e-sailing) can be extremely efficient andeffective at certain wind angles, where small amounts of addedmechanical power (e.g., electric or gas powered) can combine with thewind to extend the operational range of mobile structure 101. FIG. 5illustrates a polar comparator chart or graph 500 showing the estimatedspeed of mobile structure 101 when provided motive force by electricpower, wind, or a combination of wind and electrical power. In theembodiment shown in FIG. 5, chart 500 is generated with the assumptionthat there is a 4 knot true wind when estimating the speed of mobilestructure 101 due to motive force provided by wind (e.g., and mainsail370 and/or headsail 371) or a combination of wind and electric power.Plot 510 shows the estimated speed of mobile structure 101 generated bysuch true wind alone, as a function of relative wind angle/direction.Plot 520 (e.g., a dashed circle) shows the estimated speed of mobilestructure 101 generated by electric drive motor 230 alone and consuming750 W of power (e.g., with no wind effect). Plot 530 shows the estimatedspeed of mobile structure 101 generated by a combination of such truewind and electric drive motor 230 consuming 750 W of power.

As can be seen from chart 500, motor-sailing or e-sailing (e.g.,represented by plot 530) can generate a speed sometimes more than 25%greater than that achievable by electric power or wind alone, andgenerally greater for more than 80% of the possible relative winddirections, using the same power input. As such, controller 130 may beconfigured to use such relationship to help determine a range of mobilestructure 101 that benefits from such combinatorial effect, for example,or to determine a track for mobile structure 101 that takes advantage ofsuch relationship to increase the range of mobile structure 101, reducea rate of discharge of main battery 220, reduce a total discharge ofmain battery 220 associated with reaching a particular waypoint ordestination, increase a speed of mobile structure 101 with respect tonavigation to a desired destination, and/or otherwise adjust navigationof mobile structure 101, as described herein.

FIG. 6 shows a display view 600 including an operational range map oroverlay for mobile structure 101 in accordance with an embodiment of thedisclosure. In the embodiment presented by FIG. 1, display view 600includes navigational chart 610 including operational range map 612rendered as a shaded or colored semi-transparent area within dashed edgeline 613 on chart 610. In various embodiments, operational range map 612may be configured to show an operational range of mobile structure 101as determined based on an operational state of mobile structure 101and/or one or more environmental conditions associated with a positionand/or orientation of mobile structure 101, a planned track or route 620for mobile structure 101, and/or a planned destination 624 for mobilestructure 101. Such environmental conditions may be measured by elementsof system 100, for example, or may be received from other vessels orweather services broadcasting such environmental conditions.

In various embodiments, controller 130 of system 100 may be configuredto determine an operational state of mobile structure 101 based, atleast in part, on a desired propulsion mode selected at propulsion modeselector 650 and/or monitoring and/or sensor data provided by elementsof system 100 and/or 200; to determine environmental conditionsassociated with mobile structure 101, the determined operational stateof mobile structure 101, and/or a planned track or route 620 ordestination 624 for mobile structure 101; and to determine operationalrange map 612 based on such determined operational state and/orenvironmental conditions. Once operational range map 612 is determined(e.g., its shape and/or extent, relative to a position of mobilestructure 101 and/or an orientation of chart 610), controller 130 may beconfigured to render operational range map 612 over chart 610 on adisplay of user interface 120, for example, for display to a user tohelp facilitate route and/or passage planning for mobile structure 101.In some embodiments, controller 130 may be configured to detect that thepresent operational state of mobile structure 101 along planned route620 will result in unsafe navigation or be impossible to complete, forexample, and to autopilot mobile structure along an alternative route622 to an alternative destination 626. As such, display view 600provides a solution to autonomous or assisted route or passage planning,as described herein.

In the embodiment shown in FIG. 6, the pilot or user has used propulsionmode selector 650 to select (e.g., by user input provided to userinterface 120) a power for electric drive motor 230 at 70% of full powerwith a slider of propulsion mode selector 650, no use of generator 242,use of mainsail 370 and/or headsail 371 (e.g., for e-sailing and/orcombination of motor and wind propulsion), and reduced use of accessorypower 222 (e.g., to turn off a cooler or refrigerator associated withaccessory power sink 224). Such operational state of mobile structure101 is reflected in the size and shape of operational range map 612, asrendered over chart 610. In general, operational range map 612 may beconfigured to show straight-line range capability to a minimum desiredcharge state, which may be set by a pilot or user (in FIG. 6, theminimum charge state for main battery 220 is set to zero). If the pilotreduces the power for electric drive motor 230, then operational rangemap 612 will generally expand, and more distant destinations will beshown within reach (e.g., overlaid by operational range map 612), andthe shape of operational range map 612 will change according to theapplicable/selected sail/motor power mix, similar as shown in FIG. 5,and according to the applicable environmental conditions.

In some embodiments, controller 130 may be configured to render route620 on chart 610 as the track to the chosen destination 624 and torender state of charge (SOC) identifiers 630, 632, 634 at each waypointand, in addition, at intermediate points for relatively long tracks inbetween waypoints (e.g., shown as circular nodes along route 620 and/oralternative route 622). An estimated time of arrival (ETA) and SOCidentifier 644 may be rendered at planned destination 624. As shown inFIG. 6, display view 600 indicates that passage planning system 100estimates reaching planned destination 624 (Lymington harbor) with 5%charge remaining in main battery 220.

In various embodiments, a pilot may interrogate other waypoints oralternative route 622 and/or alternative destination 626 by selectingalternative destination 626. Upon detecting such user selection,controller 130 may be configured to render a dotted black track line toshow a recommended alternative route 622 to alternative destination 626with corresponding estimated SOCs and, with respect to alternativedestination 626, an ETA and SOC identifier 642. Alternative destination626 (Yarmouth harbor) is shown as reachable with 9% charge remaining inmain battery 220. In some embodiments, each SOC identifier may bereplaced with an ETA and SOC identifier, according to one or more userdisplay settings and/or the number of waypoints shown in display view600 (e.g., fewer waypoints and identifiers might allow larger and moreinformative identifier to be rendered without cluttering display view600 or obscuring important chart detail). In various embodiments,operational range map 612 is indicative (e.g., straight line paths arenot generally possible), but the cursor/dotted-track may be normative(e.g., system 100 may be configured to determine a best (e.g., lowestpower usage, safest traversal, quickest traversal) route to a selectedwaypoint or destination.

When determining operational range map 612, planned route 620, and/oralternative route 622, system 100 may account for wind direction andspeed, water current, sea state, and/or other factors, for example, andproject these factors forward in time, based on one or moreenvironmental forecasts (e.g., meteorological or GRIB forecasts, tidaldatabases) and/or (e.g., in case of air or sea states) a mix of forecastand measured or known environmental conditions, such as wind directionand speed (e.g., wind over tide produces stronger seas). The effect ofsea state on performance of mobile structure 101 may be provided intabular form or estimated based on a simplified analytical function, asdescribed herein.

In some embodiments, system 100 may be configured to determine and/orselect a particular power for electric drive motor 230 upon receiving auser selected destination and/or desired minimal SOC, for example, toensure reaching planned destination 624, for example, or to account forchanging and/or unpredicted environmental conditions along route 620.Thereby, system 100 is able to optimize power utilization to account forthe present and future conditions (e.g., operational state of mobilestructure 101 and/or various environmental conditions). For example, ifthe tide will turn in one hour, it may be futile to push into it; it maybe better to conserve energy for the next 45 minutes then apply a higherpower level, in order to reach planned destination 624 at a desired ETA.

In further embodiments, system 100 may be configured to determine aroute start time based on one or more user selected passage criteria.For example, a user may select to emphasize one or more of shortestjourney time, least fuel consumed, smoothest ride (e.g., avoidingwind-over-tide scenarios), a specific arrival time, a range of arrivaltimes, a combination of these, and/or other passage criteria, forexample, generally intended for selection before departure, where thejourney start-time is determined based on such criteria.

While such route or passage planning techniques are intended for usewith electric or hybrid electric (diesel/sail/electric in anycombination) watercraft, they may be applicable to any power source,including combustion engine based propulsion systems, and to terrestrialor air based mobile structures. Moreover, such techniques are notlimited to planning; similar systems and techniques may be used todynamically adjust power to electric drive motor 230, for example, tomaintain an optimal balance between sail and motor power as wind speedand/or direction changes (e.g., as described with respect to polar chart500 of FIG. 5). For example, a pilot or user may select passage criteriaincluding holding a steady 5 kts along planned route 620, and system 100may increase power to electric drive motor 230 as wind speed drops andreduce power as wind speed increases.

In some embodiments, system 100 may be configured to determine a riskprofile associated with a number of selected or otherwise determinedalternative routes (e.g., planned route 620 and alternative route 622)and render each alternative route according to a preselected colormapping to indicate one route is riskier than another. In otherembodiments, system 100 may be configured to render operational rangemap 612 according to a similar color mapping to indicate navigation toportions of operational range map 612 is riskier than navigation toother portions. In various embodiments, such color mapping may beconfigured to emphasize differentiation across a particular range ofrisk values within the risk profile, which may generally be implementedas a set of percentages (e.g., each 0-100%) linked with a particularroute, set of waypoints, and/or other sets of spatial and/or temporalpositions represented on chart 610.

Such risk profile may be based on a set of risk criteria, which may beselected by a user as part of the configuration settings or parametersfor passage planning system 100. For example, such risk criteria mayinclude risk of failing to reach a particular position withinoperational range map 612 due to known variability (or lack ofreliability) of environmental forecasts associated with a route to suchposition, risk of being stranded without electrical power at suchposition (e.g., at sea vs. at berth, determined with or withoutapplicable e-sailing capabilities), risk of delay in reaching suchposition (e.g., when the passage criteria includes a desired arrivaltime) caused by environmental conditions (e.g., tide, wind direction andstrength, measured and/or forecasted), an operational state of mobilestructure 101 (e.g., planned accessory power draw, availability of solarcharging), and/or route timing criteria (e.g., tidal gate or lockopening/closing) along the route to such position.

In various embodiments, system 100 may be configured to update the riskprofile and the rendering of alternative routes and/or operational rangemap 612 as mobile structure traverses operational range map 612. In someembodiments, system 100 may be configured to render operational rangemap 612 according to extents based on an achievable linear range ofelectric propulsion system 200, various environmental conditions, anoperational state of mobile structure 101, and a user selected maximumacceptable risk value (e.g., a type of risk criteria).

FIGS. 7-10 illustrate flow diagrams of control loops to provide passageplanning in accordance with embodiments of the disclosure. Inparticular, FIGS. 7-10 include control loops to determine a predictedspeed of mobile structure 101 based, at least in part, on applicableforecasted environmental conditions, the sailing characteristics ofmobile structure 101, and/or the assisted sailing (e.g., e-sailing)capabilities of mobile structure 101. In some embodiments, theoperations of FIGS. 7-10 may be performed by controller 130 processingand/or operating on signals received from one or more of sensors140-148, navigation control system 190, user interface 120, and/or othermodules 180. For example, in various embodiments, control loop 700(and/or other control loops of FIGS. 8-10) may be implemented and/oroperated according to any one or combination of the systems and methodsdescribed in International Patent Application No. PCT/US2014/13441 filedJan. 28, 2014 and entitled “STABILIZED DIRECTIONAL CONTROL SYSTEMS ANDMETHODS,” and/or U.S. patent application Ser. No. 14/321,646 filed Jul.1, 2014 and entitled “STABILIZED DIRECTIONAL CONTROL SYSTEMS ANDMETHODS,” each of which are hereby incorporated by reference in theirentirety.

In accordance with an embodiment, each block may be implemented entirelyas instructions executed by controller 130, for example, or may beimplemented in a combination of executable instructions and hardware,such as one or more inductors, capacitors, resistors, digital signalprocessors, and other analog and/or digital electronic devices. Itshould be appreciated that any step, sub-step, sub-process, or block ofin the control loops may be performed in an order or arrangementdifferent from the embodiment illustrated by FIGS. 7-10. For example,although control loop 720 shown in detail in FIG. 8 includes block 828,in other embodiments, block 828 may not be present, for example, and/ormay be replaced with a look up table generated through use of one ormore sensors providing corresponding measured data.

As shown in FIG. 7, control loop 700 includes speed predictor block 720providing a predicted linear/longitudinal speed (e.g., speed throughwater) for mobile structure 101. For example, speed predictor block 720may be configured to receive a time series of forecasted environmentalconditions (e.g., true wind velocities (speed and direction) 710,current velocities (speed and direction) 712) and predicted and/orselected operational states of mobile structure 101 (e.g., headings 714of mobile structure 101, mechanical propulsion powers 716), which may beassociated with a planned route for mobile structure 101, for example,and determine a time series of predicted speeds 770 of mobile structure101. As described herein, control loop 700 may be used provide predictedspeeds 770 as part of a passage planning process before mobile structure101 initiates a planned route, for example, and may also be used duringtransit along a planned route to, for example, update operational rangemap 612, update a risk profile associated with operational range map 612and/or planned route 620 and alternative route 622, or to determine orupdate any other passage state data associated with a passage plan orroute, as described herein.

As shown in FIG. 8, speed predictor block 720 may be implemented byapparent wind simulator block 820, sailing polar converter block 822,thrust and drag curve blocks 824 and 825, applied thrust predictor block828, and various other blocks configured to generate predicted speeds870 of mobile structure 101. For example, apparent wind simulator block820 may be configured to receive forecasted environmental condition andpredicted and/or selected operational state inputs 810 as shown andprovide simulated apparent wind velocities (speed and direction, as feltby/in the coordinate frame of mobile structure 101) to sailing polarconverter block 822. Sailing polar converter block 822 may be configuredto receive the simulated apparent wind velocities and provide predictedsailing speeds to thrust curve block 824. Thrust curve block 824 may beconfigured to convert the predicted sailing speeds to predicted sailingthrusts (e.g., in units of force) generated by mobile structure 101 inresponse to the predicted sailing speeds and provide the predictedsailing thrusts to combinatorial block 830.

Applied thrust predictor block 828 may be configured to convert selectedand/or predicted mechanical propulsion powers 812 and prior predictedspeeds for mobile structure 101 (e.g., as converted by unit conversionblock 826 as necessary) into predicted mechanical propulsion thrustsgenerated by mobile structure 101 in response to mechanical propulsionpowers 812 and provide the predicted mechanical propulsion thrusts tocombinatorial block 830. Drag curve block 825 may be configured toconvert prior predicted speeds for mobile structure 101 into predicteddrag thrusts generated by mobile structure 101 in response to the priorpredicted speeds and provide the predicted drag thrusts to combinatorialblock 830.

Combinatorial block 830 may be configured to combine the predictedsailing thrusts, mechanical propulsion thrusts, and drag thrusts andgenerate predicted passage thrusts, which may then be converted intopredicted speeds 870 of mobile structure 101 by force to accelerationconverter block 832, acceleration to velocity integrator block 834,and/or unit conversion block 836, as desired. In some embodiments, forceto acceleration converter block 832 may be configured to receive a timeseries of predicted masses for mobile structure 101, where the mass ofmobile structure 101 changes over the course of a planned route, such asdue to fuel consumption, for example.

In various embodiments, parameters for sailing polar converter block822, thrust and drag curve blocks 824 and 825, and/or applied thrustpredictor block 828 may be supplied by a manufacturer, for example, ormay be determined through directed sea trials (patterns of navigationalmaneuvers of mobile structure 101) or through accumulation of sensordata over time that measures the response of mobile structure 101 tovarious environmental conditions and/or operational states of mobilestructure 101. In some embodiments, thrust and drag curve blocks 824 and825 may use identical parameters, where, as shown in FIG. 8, drag curveblock 825 is driven by the predicted speed of mobile structure 101 andthrust curve block 824 is driven by predicted sailing speed generated bymobile structure 101 in response to an applicable apparent wind.

As shown in FIG. 9, apparent wind simulator block 820 may be implementedby various coordinate conversion blocks 920, 932, 942, 944, and 952, andvarious combinatorial blocks 930, 940, and 950, configured to add thevarious vectors of wind, current, and velocity of mobile structure 101(heading and speed) to generate simulated apparent wind velocities. Forexample, coordinate conversion blocks 920 may be configured to convertenvironmental condition inputs 910 into cartesian coordinates andprovide them to combinatorial block 930, which combines them andprovides the combined environmental speed effects to coordinateconversion block 932. Coordinate conversion block 932 converts theenvironmental speed effects back to polar coordinates and combinatorialblock 940 transforms them to the frame of mobile structure 101.Coordinate conversion block 942 converts the transformed environmentalspeed effects to cartesian coordinates and provides them tocombinatorial block 950. Coordinate conversion block 944 converts priorpredicted speeds of mobile structure 101 (e.g., in the frame of mobilestructure 101) to cartesian coordinates and provides them tocombinatorial block 950, which combines the prior predicted speeds andthe transformed environmental speed effects to produce apparent windvelocities in cartesian coordinates. Coordinate conversion block 952converts those cartesian apparent wind velocities into polar apparentwind velocities 960.

As shown in FIG. 10, sailing polar converter block 822 may beimplemented as executable script and/or program code configured toreceive apparent wind velocities and generate corresponding predictedsailing speeds, based, at least in part, on a true or apparent windpolar (e.g., similar to plot 510 in FIG. 5), which may then be providedto thrust curve block 824. For example, sailing polar converter block822 may be configured to determine a predicted sailing speed generatedby mobile structure 101 in response to a simulated apparent windprovided by apparent wind simulator block 820 using a true wind polar(e.g., discretized into true wind polar matrix 1002) associated withmobile structure 101. In the embodiment shown in FIG. 10, true windpolar matrix 1002 is converted to an apparent wind polar matrix in block1004 that is used (e.g., through interpolation) to determine anappropriate predicted sailing speed associated with the specific appliedapparent wind velocity.

FIG. 11 illustrates a flow diagram of a process 1100 to provide rangeestimation and/or facilitate passage planning for mobile structure 101implemented with passage planning system 100 in accordance with anembodiment of the disclosure. More generally, process 1100 may be usedto provide general navigational control for mobile structure 101. Itshould be appreciated that any step, sub-step, sub-process, or block ofprocess 1100 may be performed in an order or arrangement different fromthe embodiments illustrated by FIG. 11. For example, in otherembodiments, one or more blocks may be omitted from or added to theprocess. Furthermore, block inputs, block outputs, various sensorsignals, sensor information, calibration parameters, and/or otheroperational parameters may be stored to one or more memories prior tomoving to a following portion of a corresponding process. Althoughprocess 1100 is described with reference to systems, processes, controlloops, and images described in reference to FIGS. 1A-10, process 1100may be performed by other systems different from those systems,processes, control loops, and images and including a different selectionof electronic devices, sensors, assemblies, mobile structures, and/ormobile structure attributes, for example.

In block 1102, an operational state of and environmental conditionsassociated with a mobile structure are determined. For example,controller 130 may be configured to determine an operational state ofmobile structure 101 based on monitoring and/or sensor data provided byelements of system 100 and/or 200. In some embodiments, such operationalstate may include an electric propulsion state of system 200, a positionand/or orientation of mobile structure 101, and/or other operationalstates of mobile structure 101, as described herein. Such operationalstate may be determined based on monitoring data provided by elements ofsystems 100 and/or 200, such as a charge state for main battery 230, forexample, and/or sensor data provided by elements of system 100, such asorientation and/or position data provided by one or more of orientationsensor 140, gyroscope/accelerometer 144, GNSS 146, and/or a combinationof those, and/or other sensor data or control signals provided by otherelements of system 100. In related embodiments, controller 130 may beconfigured to determine an operational state of mobile structure 101based on a desired propulsion mode selected at propulsion mode selector650, as described herein.

In some embodiments, controller 130 may be configured to determineenvironmental conditions associated with mobile structure 101 based onsensor data provided by elements of system 100. For example, controller130 may be configured to receive a wind speed and/or direction fromspeed sensor 142, a water current speed from GNSS 146 and/or speedsensor 142, motion information from gyroscope/accelerometer 144indicating sea state, a time of day and/or sun position from GNSS 146,and/or other environmental conditions from other vessels or weatherservices through a communications module (other modules 180 and/orcommunications module 270), and such conditions may be associated with apresent and/or planned position, orientation, and/or route of mobilestructure 101.

In some embodiments, the operational state includes an electricpropulsion state, a heading, and/or a position of mobile structure 101and the environmental conditions include a wind velocity and/or acurrent velocity associated with the mobile structure and/or a plannedroute for the mobile structure. For example, controller 130 may beconfigured to determine the environmental conditions by receivingenvironmental sensor data, position data, and/or orientation data fromcorresponding sensors mounted to mobile structure 101 and/or determininga wind velocity and/or a current velocity associated with mobilestructure 101 based on the environmental sensor data, the position data,and/or the orientation data.

In other embodiments, controller 130 may be configured to determinepassage criteria associated with a planned route for mobile structure101, such as planned route 620 and/or destination 624. For example,controller 130 may be configured to receive a first portion of thepassage criteria as user input provided to user interface 120 and todetermine a second portion of the passage criteria based, at least inpart, on the first portion of the passage criteria and the operationalstate of mobile structure 101. In some embodiments, the first portion ofthe passage criteria includes a destination and a desired arrival timeand the second portion of the passage criteria comprises a series ofwaypoints defining the planned route and a corresponding series ofmechanical propulsion power levels to be provided by main battery 220 ofmobile structure 101 to propulsion system 200. More generally, passagecriteria may be determined by user selection of passage criteria and/orother parameters, as disclosed herein.

In block 1104, an operational range map is determined. For example,controller 130 may be configured to determine a size, shape, and/orextent of operational range map 612 based on the operational stateand/or environmental conditions determined in block 1102. In someembodiments, controller 130 may be configured to determine a set ofstraight line range estimations from a current position of mobilestructure 101, differentiated from each other by absolute orientation(e.g., heading), where each straight line range estimation is based on adesired speed and/or power output, an estimated wind direction andspeed, an estimated water current direction and speed, an electricpropulsion state of electric propulsion system 200, and/or otherfactors, as described herein. Controller 130 may then determine thesize, shape, and/or extent of operational range map 612 by linking theends of adjacent lines in the set of straight lines to form edge line613 and designating the area within edge line 613 and including theposition of mobile structure 101, as shown on chart 610, as operationalrange map 612.

In related embodiments, controller 130 may be configured to use positionsensor/GNSS 146 to set an initiation position for the generation ofoperational range map 612. In such embodiments, the determining theoperational range map may include determining a set of straight linerange estimations from the initiation position of mobile structure 101provided by GNSS 146, where each straight line range estimation isbased, at least in part, on a series of predicted speeds of the mobilestructure along each straight line range estimation, and where eachpredicted speed is based, at least in part, on the operational state ofthe mobile structure and a true or apparent wind polar associated withthe mobile structure, as shown and described in FIGS. 7-10.

In block 1106, an operational range map is rendered on a navigationalchart. For example, controller 130 may be configured to renderoperational range map 612 over chart 610 on a display of user interface120. In some embodiments, controller 130 may be configured to render SOCidentifiers along planned route 620, for example, and/or ETA and SOCidentifiers at planned destination 624. In related embodiments,controller 130 may be configured to receive user selection of additionalwaypoints, alternative route 622, alternative destination 626, and/orother selections corresponding to points on chart 610, for example, andto render SOC and/or ETA and SOC identifiers with such selections.Controller 130 may also be configured to render propulsion mode selector650 over a portion of chart 610 and to receive user selection of aparticular propulsion mode associated with operation of mobile structure101. Controller 130 may be configured to update the determinedoperational state of mobile structure 101 according to the selectedpropulsion mode, for example, and determine operational range map 612based on such updated operational state.

In block 1108, an operational state of a mobile structure is adjusted.For example, controller 130 may be configured to detect that planneddestination 624 is outside operational range map 612. Upon suchdetection, controller 130 may be configured to display or sound an alertto the pilot of mobile structure 101, for example, and/or to adjust anoperational state of mobile structure 101 to expand operational rangemap 612 to include planned destination 624 or to pilot mobile structure101 according to alternative route 622 to alternative destination 626that is within operational range map 612. In some embodiments,controller 130 may be an operational state controller for mobilestructure 101 and may be configured to adjust the operational state ofmobile structure 101 based, at least in part, on the operational rangemap determined in block 1104 and/or passage criteria associated with aplanned route for the mobile structure and determined in block 1102. Insuch embodiments, the operational state of mobile structure 101 mayinclude a main battery charge level for main battery 220 of propulsionsystem 200, and the adjusting the operational state of mobile structure101 may include adjusting mechanical propulsion power 221 provided bymain battery 220 to electric drive motor 230 of propulsion system 200.

In various embodiments, a planned route and/or destination may initiallybe undefined, and user selection of such route and/or destination mayoccur after operational range map 612 is rendered on a display of userinterface 120. Once such planned route or destination is selected,system 100 may be configured to determine an operational state of mobilestructure 101 to reach such planned destination at a selected ETA, forexample, and/or according to other passage criteria. If a planned routeor destination is initially outside operational range map 612, system100 may be configured to determine an updated operational state ofmobile structure 101 to expand operational range map 612 and/or adjustthe planned route to reach such planned destination, which may alsoaccount for one or more associated passage criteria. If, duringnavigation along a planned route towards a planned destination,environmental conditions change such that the planned destination isoutside an updated operational range map 612 or is otherwise unsafe,system 100 may be configured to alert the pilot of mobile structure 101and navigate to a safe anchor position, a charging station, a harborwithin the updated operational range map 612, and/or according tovarious selected and/or updated passage criteria.

Embodiments of the present disclosure can thus provide reliable andaccurate route and/or passage planning for mobile structures. Suchembodiments may be used to provide assisted and/or fully autonomousnavigation of a mobile structure and may assist in the operation ofother systems, devices, and/or sensors coupled to or associated with themobile structure, as described herein.

In accordance with additional related embodiments of the presentdisclosure, passage planning systems and methods may provide techniquesfor traversing a planned route safely using augmented reality (AR)display views, as described herein. Passage planning systems with ARdisplay views help users visualize and identify critical objects andhazards along their planned route, particularly in the context ofregulatory rules and personal preferences governing the traversal ofsuch route. For example, cardinal marks have a safe side, boats shouldpass port to port, channel marks should be respected, fishing floatsavoided, and the particular route excursion chosen to avoid such objectscan be tailored to a particular user preference. Disclosed herein aretechniques to apply maritime navigation rules and present a suggestedaction to a user in a clear simple manner, which may be accompanied byaudible alarms. Also disclosed are techniques to use such rule-basedapproach to enable a self-driving vessel.

FIGS. 12A-B illustrate diagrams of an imaging system 1210 for use with apassage planning system in accordance with an embodiment of thedisclosure. For example, imagery provided by imaging system 1210 may beused to generate AR display views, as described herein. As shown indiagram 1200 of FIG. 12A, imaging system 1210 may be mounted to the topof a mast of mobile structure 101 to provide a relatively wide field ofview (FOV) 1212 that can include the horizon and the water surfacewithin 1-2 boat lengths of mobile structure 101, as shown.

Contemporary digital cameras have so many pixels that they can combine arelatively wide FOV 1212 with good resolution (i.e., viewing range),which allows electronic stabilization to be a useful an effectivestabilization technique. However, infrared cameras, and particularlythermal cameras, typically have a smaller number of available pixels,and so it is preferable to stabilize the roll axis of the FOV and useelectronic stabilization only on the pitch axis. This makes such camerauseful for sailboats as well as power boats, and by making the camerasmall and light, it can be installed at the masthead, as shown indiagram 1200, providing a clear viewpoint even when under sail.

FIG. 12B illustrates imaging system 1210 with such mechanical rollstabilization. For example, as shown in FIG. 12B, imaging system 1210includes mounting arm 1222 and sealed housing 1224 configured to mountto the mast of mobile structure 101. Inside housing 1224 are mechanicalstabilizer 1230 and imaging module 1232. In various embodiments,mechanical stabilizer 1230 may be configured to provide mechanical rollstabilization only, and imaging system 1210 and/or controller 130 may beconfigured to provide electronic pitch and/or yaw stabilization (withrespect to the FOV of imaging system 1210). In some embodiments,mechanical stabilizer 1230 may be configured to provide mechanical rolland pitch stabilization. Mechanical stabilizer 1230 may be implementedby an articulated mount for imaging module 1232 and a motor, actuator,and/or weight, for example. Imaging module 1232 may be implemented as avisible spectrum or infrared spectrum (thermal) camera, for example, oras a combined or multi-spectrum camera. Imaging system 1210 may bepowered by solar cells or by wire to other elements of system 100, forexample, and may communicate imagery to controller 130 and/or userinterface 120 via wired and/or wireless data connections.

While it is unusual to stabilize only the roll axis mechanically, and tostabilize the pitch but not yaw axes electronically, such combination ishighly suited to maritime applications for thermal cameras and isrelatively cost efficient to manufacture (see FIG. 12B, where a motor[mechanical stabilizer 1230] provides roll rotational alignment, and thewindow does not move, and is small and so low-cost). The approach isalso valid for visible cameras, and in some applications, it may bedesirable to stabilize pitch mechanically too.

FIG. 13 shows an augmented reality (AR) display view 1300 for mobilestructure 101 with passage planning system 100 in accordance with anembodiment of the disclosure. As shown in FIG. 13, AR display view 1300has an FOV (e.g., provided by imaging system 1210) that includes sky1302, horizon 1304, and sea surface 1306 from the horizon andapproaching a perimeter of mobile structure 101. Also shown in ARdisplay view 1300 are heading indicator 1310, detector range indicator1312, and various AR markers rendered within AR display view 1300 toindicate various characteristics of recognized objects.

For example, vessel marker 1320 includes a nationality and descriptor ofthe vessel, range to the vessel, and includes a header rendered green toshow no risk of collision. Vessel marker 1322 has much the sameinformation but its header is rendered red to show there may be risk ofcollision based on the heading of that vessel. Vessel 1324 has not beenrendered with a marker because it hasn't been recognized or it issufficiently outside the planned route of mobile structure 101. Marker1330 includes similar information for a stationary drill rig but lacks acolored header because it is too far away to evaluate for collisionpurposes. Marker 1340 indicates a user-defined saved waypointcorresponding to a desired fishing area. Other markers may indicaterange to or depth of water at that point. Embodiments described hereinsupplement AR display view 1300 with additional navigationalinformation, such as navigational error alerts, safe areas associatedwith warning buoys, and/or other navigational questions. Embodimentsalso use similar techniques to autopilot mobile structure 101 and makesafety critical course changes automatically.

For example, FIGS. 14A-C show AR display views 1400, 1402, 1404 formobile structure 101 with passage planning system 100 in accordance withan embodiment of the disclosure. In FIG. 14A, AR display view 1400includes planned route indicator 1420 and navigation alert 1422providing navigation information related to buoy 1410 and vessels 1412.In FIG. 14B, AR display view 1402 includes planned route indicator 1424that has been adjusted to avoid buoy 1414. In FIG. 14C, AR display view14C includes navigation alert/shallows indicator 1426 to indicate whichside of channel buoy 1416 is not safely navigable.

In general, system 100 may be configured to generate the various ARdisplay views and safety mechanisms by (1) aggregating sensor data fromvarious sensors of system 100 into a single coherent passage databasethat is used to describe the area surrounding mobile structure 101; (2)adding navigation requests (e.g., provided to navigation control system190) to the passage database; (3) using an expert system (e.g., usingmachine learning techniques) to determine a set of potential navigationhazard contacts (e.g., collisions, groundings) based on the passagedatabase; (4) prioritizing the set of potential navigation hazardcontacts; (5) generating a set of display overlays and audible alarmsfor all potential navigation hazard contacts with hazard prioritiesabove a threshold priority; (6) autopiloting mobile structure 101 toavoid the potential navigation hazard contacts with hazard prioritiesabove the threshold priority.

For example, the sensor data may be processed by the expert system(e.g., implemented by a convolutional neural network (CNN)) to detectand classify potential navigation hazards, for example, includingvessels, buoys, diver down flags, anchor inverted triangle, and/or otherpotential navigation hazards. The passage database may include targetheadings, speeds, and/or routes, true or apparent wind velocities,coarse over ground, waypoint advance requests, for example, and may alsoinclude crowd sourced navigation data provided by other vessels. Theexpert system (e.g., implemented within or by controller 130, forexample) may be configured to determine potential navigation hazardcontacts based on all detected vessel's velocities, as well as requestedchanges to the velocity of mobile structure 101. In some embodiments,such expert system may be configured to project other vessel's motionunder the assumption that their motion is constant to first (constantvelocity) or second (constant acceleration) order. In relatedembodiments, applicable regulatory rules (e.g., “internationalregulations for preventing collisions at sea 1972 (COLREGs)”) may beapplied to help project other vessel's motion. For example, two poweredvessels heading directly towards each other should both turn tostarboard and pass port to port and so have predictable evolving routes.

In various embodiments, controller 130 (e.g., the expert system) may beconfigured to prioritize each determined potential navigation hazardcontact according to immediacy and severity. Immediacy is defined asinversely proportional to the time to reach the potential navigationhazard contact (e.g., impact in the case of collision, shallow thresholdcrossing, or crossing a line of channel marks, etc.). Severity isdefined as proportional to risk of property damage or human injury;impact with another vessel has higher severity than entering shallowwater (where grounding is not imminent). According to such strategy,rapidly approaching vessels or vessels behaving unusually may be flaggedas potential navigation hazard contacts from further away, creating ahigher level of severity.

The expert system may be configured to determine when to show warningsand which views to show, without the user having to intervene.Similarly, the expert system may be configured to choose display viewsand apps according to context (e.g., showing docking views and enteringassisted docking mode when close to a dock). The highest prioritypotential navigation hazard contacts trigger a set of display overlaysand audible alarms, and system 100 displays the highest prioritypotential navigation hazard contacts to the user. Alarm characteristicsare chosen to reflect the severity and urgency, with increasing volumeand cadence as the immediacy and severity increase. If system 100 isautopiloting mobile structure 101, after alarming for a maximum inactionperiod (e.g., 20 s), system 100 may be configured to adjust a headingand/or speed of mobile structure 101 to avoid the potential navigationhazard contact. For example, a target heading may be adjusted, then oncepast a potential navigation hazard contact, restored, to route mobilestructure 101 around a fishing float. The direction of turn may beselected or determined to minimize risk (collision with other vessels,shallow water, etc.).

CNNs often operate most efficiently with fixed frame sizes (e.g.,512×512), yet maritime imagery often needs lots of pixels for reliablemonitoring. This means that a CNN might need to process 100 s of fixedframes to process a full 360 degree maritime scene, and even then, theCNN would likely be confused by large objects (e.g., a single vesselclose-up, could be larger than 512×512). Detailed (distant) objects tendto be close to the horizon, so for many maritime applications it is onlynecessary to process a strip just at or below the horizon to reliablydetect and classify the fine pixelated far off objects. Big objectsclose-up can be processed by the CNN by down sampling.

FIG. 15 shows a region-differentiated image processing strategy 1500 forAR display views for mobile structure 101 with passage planning system100 in accordance with an embodiment of the disclosure. For example, asshown in FIG. 15, region-differentiated image processing strategy 1500includes a first large region 1510 that encompasses the entirety of theFOV of imaging system 1210, for example, and a number of (e.g., seven)additional smaller overlapping regions 1520 distributed across thehorizon in the FOV of imaging system 1210, as shown, each being the samepixel size as the applicable CNN. In some embodiments, system 100 may beconfigured to downsample full frame region 1510 to a specified fixedframe size (e.g., using decimation, bicubic downsampling, etc.)associated with a CNN implemented by controller 130, for example, andthe result of such processing can be detection and classification ofrelatively large port channel buoy 1530. System 100 may then identifyand process horizon regions 1520 (e.g., using horizon/contrast detectionand a desired overlap distribution), and the result of such processingcan be the detection and classification of relatively small objects,such as buoy 1532 and powerboat 1534. Embodiments using image processingstrategy 1500 or similar are able to significantly reduce the computingresources necessary to provide reliable object detection andclassification, as described herein.

Embodiments of system 100 integrated with imaging system 1210 may beused to create a visual navigation log, for example, which could be usedfor insurance purposes and/or monitored by a rental company tocontemporaneously analyze detected near-misses or other dangerousbehavior over the rental period. Similar image processing techniques canbe used to implement search and rescue systems, where feeds into thedatabase include man over board location, vessel-in-trouble location,search area, search pattern, etc. Such imagery feed can include a livelink to coastguard or border protection services or craft. Embodimentscan be used to implement fishing applications, where desirable fishingspots or waypoints are included in the passage database, along withthermocline, water temperature, and/or other sensor data. Embodimentscan also be used to implement sailboat racing applications, such thatthe passage database can include race marks, wind information (which canbe visualized in 2 d or 3 d), live positions of competing ships, watercurrents, etc. Specialized racing rules can be included in the database(similar to the regulatory rules) so that system 100 indicates throughan AR display view whether a pilot should give or is due water at a racemark, or if a pilot has broken a rule and must perform penalty turns.

FIG. 16 illustrates a flow diagram of a process 1600 to provide ARdisplay views for mobile structure 101 in accordance with an embodimentof the disclosure. It should be appreciated that any step, sub-step,sub-process, or block of process 1600 may be performed in an order orarrangement different from the embodiments illustrated by FIG. 16. Forexample, in other embodiments, one or more blocks may be omitted from oradded to the process. Furthermore, block inputs, block outputs, varioussensor signals, sensor information, calibration parameters, and/or otheroperational parameters may be stored to one or more memories prior tomoving to a following portion of a corresponding process. Althoughprocess 1600 is described with reference to systems, processes, controlloops, and images described in reference to FIGS. 1A-15, process 1600may be performed by other systems different from those systems,processes, control loops, and images and including a different selectionof electronic devices, sensors, assemblies, mobile structures, and/ormobile structure attributes, for example.

In block 1602, ranging sensor data from a mobile structure is aggregatedinto a passage database. For example, controller 130 may be configuredto aggregate perimeter sensor data provided by perimeter ranging system148 into a passage database, as described herein. In some embodiments,the passage database may include, in addition to the perimeter sensordata, a regulatory navigation ruleset (e.g., local, territorial,international), user navigation preferences (e.g., a navigation comfortmode, navigational aggressiveness, a minimum priority threshold, and/orother user navigation preferences), passage criteria associated with aplanned route for the mobile structure (e.g., a planned destination,time of arrival, route, and/or a minimum SOC), and/or navigation controlsignals (e.g., a time series of navigation control signals) provided tonavigation control system 190 of mobile structure 101 (e.g., by userinterface 120 and/or controller 130).

In some embodiments, controller 130 may be configured to additionally oralternatively aggregate environmental sensor data provided by one ormore environmental sensors mounted to or within mobile structure 101into the passage database. In related embodiments, the passage databasemay include a fishing compendium comprising a variety of fishing-relateddata, such as positions of fishing locations, seasonal and speciesfishing catch rates, suggested baits or lures, correlations of catchrates to local time of day, weather (e.g., sun or cloud cover), seasurface temperatures, wind, tide, lunar or solar activity, and/or otherenvironmental conditions.

In block 1604, potential navigation hazard contacts are determined. Forexample, controller 130 may be configured to determine a set ofpotential navigation hazard contacts based, at least in part, on thepassage database updated in block 1602. In some embodiments, controller130 may be configured to detect and classify one or more navigationhazards within a preselected range of mobile structure 101 (e.g., thesensing range of an associated ranging sensor system) based, at least inpart, on the ranging sensor data in the passage database. Controller 130may then identify at least one of the one or more navigation hazards asthe set of potential navigation hazard contacts based, at least in part,on a projected course of mobile structure 101 (e.g., a target heading,target speed, and/or a target or planned route for mobile structure 101)and a position and/or a projected motion of the at least one navigationhazard. Such projected motion may assume a detected velocity oracceleration of the at least one navigation hazard be constant over thetime period of the projection.

In some embodiments, the ranging sensor system includes imaging system1210 implemented as an embodiment of perimeter ranging system 148. Insuch embodiments, controller 130 may be configured to process imageryprovided by imaging system 1210 according to region-differentiated imageprocessing strategy 1500 using a fixed frame CNN characterized by afixed frame pixel size that is smaller than a frame pixel size of theimagery provided by the imaging system, as described herein.

In additional or alternative embodiments, where the passage databasecomprises a fishing compendium, controller 130 may be configured todetermine a set of potential fishing locations based, at least in part,on the passage database updated in block 1602. Such set of potentialfishing locations may be selected or identified within the passagedatabase based on an achievable navigational range of mobile structure101, for example, or desired time of arrival at a destination (e.g., toreturn to a berth).

In block 1606, hazard priorities for potential navigation hazardcontacts are determined. For example, controller 130 may be configuredto determine a set of hazard priorities corresponding to the set ofpotential navigation hazard contacts determined in block 1604 based, atleast in part, on the passage database updated in block 1602. In someembodiments, controller 130 may be configured to determine an immediacyfor each potential navigation hazard contact based, at least in part, onan estimated time for the mobile structure to reach the correspondingnavigation hazard; determine a severity for each potential navigationhazard contact based, at least in part, on a size, shape, velocity,and/or type of the corresponding navigation hazard; and determine theset of hazard priorities based on a combination (e.g., sum) of theimmediacy and severity for each potential navigation hazard contact inthe corresponding set of potential navigation hazard contacts.

In additional or alternative embodiments, where the passage databasecomprises a fishing compendium, controller 130 may be configured todetermine a set of fishing location priorities corresponding to the setof potential fishing locations determined in block 1604 based, at leastin part, on the passage database updated in block 1602.

In block 1608, user alerts for potential navigation hazard contacts aregenerated. For example, controller 130 may be configured to generate auser alert for each potential navigation hazard contact based, at leastin part, on its corresponding priority and a preselected minimumpriority threshold. In some embodiments, when generating a user alert,controller 130 may be configured to render a navigation alertidentifying one or more potential navigation hazard contacts withcorresponding hazard priorities above the preselected minimum prioritythreshold on a display of user interface 120 associated with mobilestructure 120. In other embodiments, controller 130 may be configured togenerate an audible alert via a speaker of user interface 120 for atleast one of the one or more potential navigation hazard contacts withcorresponding hazard priorities above the preselected minimum prioritythreshold.

In additional or alternative embodiments, where the passage databasecomprises a fishing compendium, controller 130 may be configured torender navigation information or identifiers identifying one or morepotential fishing locations with corresponding fishing locationpriorities above a preselected minimum fishing priority threshold on adisplay of user interface 120 associated with mobile structure 120.Controller 130 may also be configured to generate an audible alert via aspeaker of user interface 120 for at least one of the one or morepotential fishing locations with corresponding fishing locationpriorities above the preselected minimum fishing priority threshold.

In block 1610, an operational state of a mobile structure is adjusted.For example, controller 130 may be configured to adjust, usingoperational state controller coupled to or within mobile structure 101(e.g., an embodiment of controller 130), the operational state of mobilestructure 101 based, at least in part, on the set of potentialnavigation hazard contacts determined in block 1604, the correspondingset of hazard priorities determined in block 1606, and a preselectedminimum priority threshold. In some embodiments, such operational stateof mobile structure 101 may include a target heading, speed, and/orroute for mobile structure 101, and controller 130 may be configured toadjust the operational state of mobile structure 101 by adjusting thetarget heading, speed, and/or route for mobile structure 101 to avoidone or more potential navigation hazard contacts with correspondinghazard priorities above the preselected minimum priority threshold.

In additional or alternative embodiments, where the passage databasecomprises a fishing compendium, controller 130 may be configured toadjust a target heading, speed, and/or route for mobile structure 101 tonavigate to one or more potential fishing spots with correspondingfishing location priorities above a preselected minimum fishing prioritythreshold.

Embodiments of the present disclosure can thus provide reliable andaccurate AR display views for mobile structures and/or passage planningsystems. Such embodiments may be used to assist in the operation ofvarious systems, devices, and/or sensors coupled to or associated withmobile structure 101, such as navigation control system 190 in FIG. 1A,as described herein. Moreover, such embodiments may be employed toreliably and accurately provide situational awareness and collisiondetection and avoidance, as described herein.

As noted herein, embodiments of imaging systems to be used as perimeterand/or ranging sensors, such as for passage planning, collisionavoidance, and general marine navigation, may advantageously beimplemented with mechanical roll stabilization and electronic (e.g.,image clipping) pitch stabilization. In general, imaging systems mayenclose all elements within a waterproof housing, so its imaging modulerotates and the window through which the imaging module views a scenestays still. Such enclosure simplifies manufacture and maintenance andmakes the imaging system far more robust. In particular, thermal imagingmodules typically do not work well with electronic stabilization methodsin roll and pitch because there are insufficient pixels (e.g., for asensor that would be mounted to a sailing ship). As such, mechanicalroll stabilization with digital pitch windowing is especially wellsuited to thermal imagery, more so because (relatively expensive) IR orthermal enclosure windows, as described herein, can be relatively smalland fixed. Moreover, there is typically limited space at masthead andembodiments may be formed relatively compactly.

FIGS. 17A-B illustrate diagrams of imaging systems 1210 and 1710 for usewith a passage planning system 100 in accordance with an embodiment ofthe disclosure. More generally, imaging systems 1210 and 1710 may beused with a general navigation system 100 configured to provideperimeter monitoring, autopiloting, and/or collision avoidance as mobilestructure 101 proceeds along a route. For example, imaging systems 1210and 1710 may be used to generate AR display views, as described herein.As shown in FIG. 17A, imaging system 1210 (e.g., a single elementimaging system) includes mounting arm 1222, sealed enclosure/housing1224, and mounting base and/or I/O interface 1223 configured to mount tothe mast of mobile structure 101. Inside sealed enclosure 1224 are amechanical stabilizer and an imaging module. In various embodiments, themechanical stabilizer may be configured to provide mechanical rollstabilization only, and imaging system 1210 and/or controller 130 may beconfigured to provide electronic pitch and/or yaw stabilization (withrespect to the FOV of imaging system 1210).

In alternative embodiments, the mechanical stabilizer may be configuredto provide mechanical roll and pitch stabilization. The mechanicalstabilizer may be implemented by an articulated mount for the imagingmodule and a motor, actuator, and/or weight, for example. The imagingmodule inside sealed enclosure 1224 may be implemented as a visiblespectrum or infrared spectrum (thermal) camera, for example, or as acombined or multi-spectrum camera. Imaging system 1210 may be powered bysolar cells or by wire (e.g., through mounting base 1223) to otherelements of system 100, for example, and may communicate imagery/imagedata to controller 130 and/or user interface 120 via wired and/orwireless data connections.

Although shown with a straight mounting arm 1222, mounting arm 1222 maybe longer or shorter or curved to as to provide room for other sensorsand devices mounted to a masthead, for example. In the embodiment shownin FIG. 17A, imaging system 1210 is approximately 9 inches by 4 inchesby 4 inches and is relatively light (every pound added to the top of amast creates a counter moment at the keel due to the length of themast).

As noted herein, while it is unusual to stabilize only the roll axismechanically, and to stabilize the pitch but not yaw axeselectronically, such combination is highly suited to maritimeapplications for thermal cameras and is relatively cost efficient tomanufacture (see FIG. 12B, where a motor [mechanical stabilizer 1230]provides roll rotational alignment, and the window does not move, and issmall and so low-cost). The approach is also valid for visible cameras,and in some applications, it may be desirable to stabilize pitchmechanically too.

As shown in FIG. 17B, imaging system 1710 includes window 1725, sealedenclosure/dome 1724, and mounting base and/or I/O interface 1722, in theform of a marine navigation light, configured to mount to the mast ofmobile structure 101. Inside sealed enclosure 1724 are a mechanicalstabilizer and an imaging module. In various embodiments, the mechanicalstabilizer may be configured to provide mechanical roll stabilizationonly, and imaging system 1710 and/or controller 130 may be configured toprovide electronic pitch and/or yaw stabilization (with respect to theFOV of imaging system 1710), as described herein.

FIG. 18 illustrates a diagram of a dual element imaging system 1810 foruse with a passage planning system and/or a general navigation system100 in accordance with an embodiment of the disclosure. For example,dual element imaging system 1810 may be used to generate AR displayviews, as described herein. As shown in FIG. 18, dual element imagingsystem 1810 includes sealed enclosure 1811 with back 1812 and sealedface 1814 configured to support two optical windows aimed along anoptical axis of dual element imaging system 1810. In particular, dualelement imaging system 1810 includes dual element mechanical rollstabilizer 1820 configured to provide mechanical roll stabilization offirst and second imaging modules within sealed enclosure 1811simultaneously with respect to the optical axis of dual element imagingsystem 1810. For example, sealed face 1814 may support a visiblespectrum window 1832 and an infrared window 1833, where each windowforms a weatherproof seal with sealed face 1814 of sealed enclosure 1811and is configured to be substantially transparent to visible spectrumlight and infrared (or thermal) radiation, respectively. In variousembodiments, dual element mechanical roll 1820 stabilizer may beconfigured to provide mechanical roll stabilization of multiple imagingmodules simultaneously, but only roll stabilization, and dual elementimaging system 1810 and/or controller 130 may be configured to provideelectronic pitch and/or yaw stabilization (with respect to the FOVs ofdual element imaging system 1810).

FIG. 19 illustrates a diagram of a dual element imaging system enclosure1911 for use with a passage planning system and/or a general navigationsystem 100 in accordance with an embodiment of the disclosure. Forexample, dual element imaging system enclosure 1911 may form anenclosure for dual element imaging system 1810, as described herein. Asshown in FIG. 18, dual element imaging system enclosure 1911 includesenclosure wall 1912, sealed cavity 1913, and mounting base 1916. Invarious embodiments, power and/or data signaling may be routed throughback 1812 of sealed enclosure 1811 or through mounting base 1916, asdescribed herein. In some embodiments, such power and/or datasignalizing interface or communications link may be implementedaccording to a power-over-Ethernet protocol and/or specification. Alsoshown in FIG. 19 are representations of visible spectrum imaging module1932 and thermal imaging module 1933. In various embodiments, dualelement imaging system enclosure 1911 may be formed from metal toprovide thermal management of the electronics and other elements of dualelement imaging system 1810.

FIGS. 20-21 show imaging system assemblies 2000, 2100 for dual elementimaging system 1810 for use with a passage planning system and/or ageneral navigation system 100 in accordance with an embodiment of thedisclosure. For example, imaging system assemblies 2000, 2100 may beused to capture and provide roll stabilized imagery for AR displayviews, as described herein. In FIG. 20, imaging system assembly 2000(e.g., a dual element imaging system assembly) includes supportstructure 2011, interface module 2016, power module 2015, controllermodule 2018, infrared or thermal imaging module 2033, and visiblespectrum imaging module 2032. In various embodiments, controller module2018 may be implemented similarly to controller 130 and/or otherelements of system 100 mounted to a printed circuit board (PCB), forexample, and may be configured to control operation of imaging systemassembly 2000, such as by performing any of the methods, processes, andcontrol loops described herein. Interface module 2016 may also includevarious logic and analog devices mounted to a PCB be configured toreceive cabling to form a wired communications link with elements ofsystem 100, for example, to receive power, and/or to form one or morewireless communications links, as described herein. Power module 2015may also include various logic and/or analog devices, including powerregulation circuit elements, and may be configured to receive systempower from interface module 2016 and provide regulated and/or filteredpower to various elements of imaging system assembly 2000.

Also shown in FIG. 20 is a portion of belt 2022 configured to facilitateoperation of a dual element mechanical roll stabilizer depicted in moredetail in FIG. 21. Belt 2022 may be used to effect and synchronize rollcompensation of both infrared/thermal imaging module 2033 and visiblespectrum imaging module 2032 using a single actuator or motor, asdescribed herein.

In FIG. 21, imaging system assembly 2100 similarly includes supportstructure 2011, interface module 2016, power module 2015, controllermodule 2018, infrared/thermal imaging module 2033, visible spectrumimaging module 2032, and belt 2022, and additional shows dual elementmechanical roll stabilizer mechanism 2122 and dual element mechanicalroll stabilizer actuator 2100. In various embodiments, controller module2018 may generate and provide control signals to actuator 2120 toprovide mechanical roll stabilization of imaging modules 2032-2033simultaneously with respect to the optical axis of imaging systemassembly 2100. In particular, actuator 2120, which may be implemented asa fast stepping motor, may drive belt 2022, which in turn drives dualelement mechanical roll stabilizer mechanism 2122. dual elementmechanical roll stabilizer mechanism 2122 is coupled to each of imagingmodules 2032-2033 and is configured to adjust their boresight rolls(e.g., their respective rotations about their individual optical axes,which are aligned roughly parallel to the optical axis of dual elementimaging system 1810/imaging system assembly 2100 so that imaging modules2032-2033 share at least a partial overlapping FOV).

FIGS. 22A-B show AR display views 2200, 2202 including imagery/imagedata provided by dual element imaging system 1810 for mobile structure101 with a passage planning system and/or a general navigation system100 in accordance with an embodiment of the disclosure. For example, inFIG. 22A, AR display view 2200 includes thermal image 2233A (e.g.,captured by thermal imaging module 2033) side by side withcontemporaneous visible spectrum image 2232A (e.g., captured by visiblespectrum imaging module 2032), where both images share at least apartial FOV of the open ocean while mobile structure 101 is underway. InFIG. 22B, AR display view 2202 includes thermal image 2233B (e.g.,captured by thermal imaging module 2033) side by side withcontemporaneous visible spectrum image 2232B (e.g., captured by visiblespectrum imaging module 2032), where both images share at least apartial FOV of a port while mobile structure 101 is navigating throughthe port. Although shown in FIGS. 22A and 22B as separate images, inother embodiments, thermal and visible spectrum images may be combined(e.g., registered, scaled, and/or rotated) so as to form a combinedimage with features derived from the thermal images and the visiblespectrum images.

FIGS. 23A-B show AR display views 2300, 2302 for mobile structure 101with a passage planning system in accordance with an embodiment of thedisclosure. For example, in FIG. 23A, AR display view 2300 includeschart display 2320 side by side with AR display 2330. In general, ARdisplay 2330 includes features similar to those found in AR displayviews 1300 and 1400-1404 of FIGS. 13-14C. More specifically, AR displayview 2330 includes image data 2332/2333 overlaid with various AR objectmarkers 2336 and AR planned route indicator 2334, each of which havecorresponding features rendered within chart display 2320, includingplanned route indicator 2324 and various object indicators 2326. Inaddition, chart display 2320 includes FOV indicator 2328 illustrating amonitoring area and/or the extents of the FOV associated with image data2332/2333 of AR display 2330.

In FIG. 23B, AR display view 2302 includes chart display 2340 side byside with AR display 2350. AR display view 2350 includes image data2332/2333 overlaid with various AR object markers 2356, AR planned routeindicator 2354, and AR monitoring perimeter marker 2352, each of whichhave corresponding features rendered within chart display 2340,including planned route indicator 2344, various object indicators 2346,and monitoring perimeter indicator 2342.

FIG. 24 illustrates a flow diagram of a process 2400 to provide rollstabilized imagery for AR display views for mobile structure 101 with apassage planning system and/or a general navigation system 100 inaccordance with an embodiment of the disclosure. It should beappreciated that any step, sub-step, sub-process, or block of process2400 may be performed in an order or arrangement different from theembodiments illustrated by FIG. 24. For example, in other embodiments,one or more blocks may be omitted from or added to the process.

Furthermore, block inputs, block outputs, various sensor signals, sensorinformation, calibration parameters, and/or other operational parametersmay be stored to one or more memories prior to moving to a followingportion of a corresponding process. Although process 2400 is describedwith reference to systems, processes, control loops, and imagesdescribed in reference to FIGS. 1A-23B, process 2400 may be performed byother systems different from those systems, processes, control loops,and images and including a different selection of electronic devices,sensors, assemblies, mobile structures, and/or mobile structureattributes, for example.

In block 2402, orientation data from an orientation sensor coupled to amobile structure is received. For example, controller 130 and/orcontroller module 2018 may be configured to receive orientation data(e.g., roll, pitch, and/or yaw) indicating an orientation of mobilestructure 101 from orientation sensor 140. In some embodiments, imagingsystem 1210, 1710, and/or 1810 may be implemented with an orientationsensor similar to orientation sensor 140 that is disposed within thesealed enclosure of the imaging system, for example, and controller 130and/or controller module 2018 may be configured to receive orientationdata from such integrated sensor. In related embodiments, controller 130and/or controller module 2018 may be configured to detect a horizon inimage data provided by imaging modules of imaging system 1210, 1710,and/or 1810 and determine orientation data corresponding to anorientation of one or more imaging modules of imaging system 1210, 1710,and/or 1810, where the imaging modules are used effectively as rollorientation sensors providing roll orientation data.

In block 2404, a boresight roll of an imaging system coupled to a mobilestructure is determined. For example, controller 130 and/or controllermodule 2018 may be configured to determine a boresight roll of imagingsystem 1210, 1710, and/or 1810 coupled to mobile structure 101 based, atleast in part, on the orientation data received in block 2402. In someembodiments, imaging system 1210, 1710, and/or 1810 may be mounted tomobile structure 101 such that the roll of mobile structure 101substantially equals the boresight roll of imaging system 1210, 1710,and/or 1810, and so the roll of mobile structure 101 (e.g., relative toa reference or at rest orientation) may be used directly as theboresight roll of imaging system 1210, 1710, and/or 1810. Similarly, ifthe orientation sensor is integrated with imaging system 1210, 1710,and/or 1810, or based on image data provided by imaging modules ofimaging system 1210, 1710, and/or 1810, the orientation data may be useddirectly as the boresight roll of imaging system 1210, 1710, and/or1810.

In other embodiments, the boresight of imaging system 1210, 1710, and/or1810 may not be parallel to the longitudinal axis of mobile structure101, or the orientation sensor may not be aligned with the boresight ofimaging system 1210, 1710, and/or 1810 and controller 130 and/orcontroller module 2018 may be configured to perform a coordinate frametransformation on such orientation data (e.g., from orientation sensor140) to determine a boresight roll of imaging system 1210, 1710, and/or1810 from the orientation data.

In block 2406, a mechanical roll stabilizer of an imaging system may becontrolled to compensate for a boresight roll. For example, controller130 and/or controller module 2018 may be configured to controlmechanical roll stabilizer 1820 of imaging system 1810 to compensate forthe boresight roll determined in block 2406. In some embodiments,controller module 2018 may be configured to generate a control signalbased, at least in part, on the boresight roll determined in block 2404and provide the control signal to actuator 2120 of mechanical rollstabilizer 1820 to rotate imaging modules 2032 and 2033 to compensatefor the determined boresight roll.

In other embodiments, controller 130 may be configured to generate acontrol signal based, at least in part, on the boresight roll determinedin block 2404 and provide the control signal to controller module 2018of imaging system 1810. Controller module 2018 may be configured toreceive the control signal from controller 130 and control actuator 2120of mechanical roll stabilizer 1820, based, at least in part, on thereceived control signal, to rotate imaging modules 2032 and 2033 tocompensate for the determined boresight roll. In further embodiments,controller 130 may be configured to provide the boresight rolldetermined in block 2404 to controller module 2018 of imaging system1810. Controller module 2018 may be configured to receive the determinedboresight roll from controller 130, generate a control signal based, atleast in part, on the determined boresight roll, and provide the controlsignal to actuator 2120 of mechanical roll stabilizer 1820 to rotateimaging modules 2032 and 2033 to compensate for the determined boresightroll.

In various embodiments, mechanical roll stabilizer 1820 is coupled toimaging modules 2032 and 2033 and configured to provide mechanical rollstabilization of imaging modules 2032 and 2033 simultaneously withrespect to the optical axis of imaging system 1810. In additionalembodiments, controller 130 and/or controller module 2018 may beconfigured to determine and/or control a mechanical roll stabilizer ofimaging system 1210, 1710, and/or 1810 to apply a reference boresightroll to imaging modules of imaging system 1210, 1710, and/or 1810 toalign the imaging modules substantially to the horizon while mobilestructure 101 is at rest. Once the imaging modules are roll stabilized,imagery from the imaging modules may be transmitted to controller 130 tobe processed into various display views, including AR display views, asdescribed herein. For example, with respect to process 1600 of FIG. 16,the resulting mechanically roll-stabilized image data may be used as theranging sensor data identified in block 1602 and aggregated into apassage database, which may then be used to proceed through theremaining block of process 1600, as described herein.

Embodiments of the present disclosure can thus provide reliable andaccurate AR display views for mobile structures and/or passage planningsystems. Such embodiments may be used to assist in the operation ofvarious systems, devices, and/or sensors coupled to or associated withmobile structure 101, such as navigation control system 190 in FIG. 1A,as described herein. Moreover, such embodiments may be employed toreliably and accurately provide situational awareness and collisiondetection and avoidance, as described herein.

Where applicable, various embodiments provided by the present disclosurecan be implemented using hardware, software, or combinations of hardwareand software. Also, where applicable, the various hardware componentsand/or software components set forth herein can be combined intocomposite components comprising software, hardware, and/or both withoutdeparting from the spirit of the present disclosure. Where applicable,the various hardware components and/or software components set forthherein can be separated into sub-components comprising software,hardware, or both without departing from the spirit of the presentdisclosure. In addition, where applicable, it is contemplated thatsoftware components can be implemented as hardware components, andvice-versa.

Software in accordance with the present disclosure, such asnon-transitory instructions, program code, and/or data, can be stored onone or more non-transitory machine readable mediums. It is alsocontemplated that software identified herein can be implemented usingone or more general purpose or specific purpose computers and/orcomputer systems, networked and/or otherwise. Where applicable, theordering of various steps described herein can be changed, combined intocomposite steps, and/or separated into sub-steps to provide featuresdescribed herein.

Embodiments described above illustrate but do not limit the invention.It should also be understood that numerous modifications and variationsare possible in accordance with the principles of the invention.Accordingly, the scope of the invention is defined only by the followingclaims.

What is claimed is:
 1. A method comprising: receiving orientation datafrom an orientation sensor coupled to a mobile structure; determining aboresight roll of an imaging module of an imaging system coupled tomobile structure based, at least in part, on the received orientationdata, wherein an imaging module is disposed within a sealed enclosureand configured to capture image data along an optical axis of theimaging system according to a field of view (FOV) of the imaging module;and controlling a mechanical roll stabilizer of the imaging systemcoupled to the imaging module to compensate for the determined boresightroll, wherein the mechanical roll stabilizer is disposed within thesealed enclosure and is configured to provide mechanical rollstabilization of the imaging module with respect to the optical axis ofthe imaging system.
 2. The method of claim 1, wherein the controllingthe mechanical roll stabilizer comprises: generating a control signalbased, at least in part, on the determined boresight roll and providingthe control signal to an actuator of the mechanical roll stabilizer torotate the imaging module to compensate for the determined boresightroll.
 3. The method of claim 1, wherein: the controlling the mechanicalroll stabilizer comprises generating a control signal based, at least inpart, on the determined boresight roll and providing the control signalto a controller module of the imaging system; and the controller moduleof the imaging system is configured to receive the control signal fromthe controller of the mobile structure and control an actuator of themechanical roll stabilizer, based, at least in part, on the receivedcontrol signal, to rotate the imaging module to compensate for thedetermined boresight roll.
 4. The method of claim 1, wherein: thecontrolling the mechanical roll stabilizer comprises providing thedetermined boresight roll to a controller module of the imaging system;and the controller module of the imaging system is configured to receivethe determined boresight roll from the controller of the mobilestructure, generate a control signal based, at least in part, on thedetermined boresight roll, and provide the control signal to an actuatorof the mechanical roll stabilizer to rotate the imaging module tocompensate for the determined boresight roll.
 5. The method of claim 1,wherein the imaging module comprises a first imaging module configuredto capture thermal image data according to a first FOV of the firstimaging module, and wherein the imaging system further comprises: asecond imaging module disposed within the sealed enclosure andconfigured to capture visible spectrum image data along the optical axisof the imaging system according to a second FOV of the second imagingmodule; wherein the mechanical roll stabilizer is coupled to the firstand second imaging modules and configured to provide mechanical rollstabilization of the first and second imaging modules simultaneouslywith respect to the optical axis of the imaging system.
 6. The method ofclaim 1, further comprising: aggregating image data provided by theimaging system mounted to the mobile structure into a passage database;determining a set of potential navigation hazard contacts based, atleast in part, on the passage database; and determining a set of hazardpriorities corresponding to the set of potential navigation hazardcontacts based, at least in part, on the passage database.
 7. The methodof claim 6, wherein the determining the set of potential navigationhazard contacts comprises: detecting and classifying one or morenavigation hazards within a preselected range of the mobile structurebased, at least in part, on the passage database; and identifying atleast one of the one or more navigation hazards as the set of potentialnavigation hazard contacts based, at least in part, on a projectedcourse of the mobile structure and a position and/or a projected motionof the at least one navigation hazard.
 8. The method of claim 6, whereinthe determining the set of potential navigation hazard contactscomprises: processing the image data provided by the imaging systemaccording to a region-differentiated image processing strategy using afixed frame convolutional neural network (CNN) characterized by a fixedframe pixel size that is smaller than a frame pixel size of the imagedata provided by the imaging system.
 9. The method of claim 6, whereinthe determining the set of hazard priorities comprises: determining animmediacy for each potential navigation hazard contact based, at leastin part, on an estimated time for the mobile structure to reach thecorresponding navigation hazard; determining a severity for eachpotential navigation hazard contact based, at least in part, on a size,shape, velocity, and/or type of the corresponding navigation hazard; anddetermining the set of hazard priorities based on a combination of theimmediacy and severity for each potential navigation hazard contact inthe corresponding set of potential navigation hazard contacts.
 10. Themethod of claim 6, further comprising: generating a user alert for eachpotential navigation hazard contact based, at least in part, on itscorresponding priority and a preselected minimum priority threshold;wherein the generating the user alert comprises rendering a navigationalert identifying one or more potential navigation hazard contacts withcorresponding hazard priorities above the preselected minimum prioritythreshold on a display of a user interface associated with the mobilestructure and/or generating an audible alert via a speaker of the userinterface for at least one of the one or more potential navigationhazard contacts with corresponding hazard priorities above thepreselected minimum priority threshold; and adjusting, using anoperational state controller coupled to or within the mobile structure,the operational state of the mobile structure based, at least in part,on the set of potential navigation hazard contacts, the correspondingset of hazard priorities, and a preselected minimum priority threshold;wherein the operational state of the mobile structure comprises a targetheading, speed, and/or route for the mobile structure; and wherein theadjusting the operational state of the mobile structure comprisesadjusting the target heading, speed, and/or route for the mobilestructure to avoid one or more potential navigation hazard contacts withcorresponding hazard priorities above the preselected minimum prioritythreshold.
 11. An imaging system configured to be coupled to a mobilestructure, the imaging system comprising: an imaging module disposedwithin a sealed enclosure and configured to capture image data along anoptical axis of the imaging system according to a field of view (FOV) ofthe imaging module; a mechanical roll stabilizer coupled to the imagingmodule, disposed within the sealed enclosure, and configured to providemechanical roll stabilization of the imaging module with respect to theoptical axis of the imaging system; and a logic device configured tocommunicate with the imaging system, wherein the logic device isconfigured to: receive orientation data from an orientation sensorcoupled to the mobile structure; determine a boresight roll of theimaging module based, at least in part, on the received orientationdata; and control the mechanical roll stabilizer to compensate for thedetermined boresight roll.
 12. The imaging system of claim 11, whereinthe logic device comprises a controller module of the imaging systemthat is disposed within the sealed enclosure, and wherein thecontrolling the mechanical roll stabilizer comprises: generating acontrol signal based, at least in part, on the determined boresight rolland providing the control signal to an actuator of the mechanical rollstabilizer to rotate the imaging module to compensate for the determinedboresight roll.
 13. The imaging system of claim 11, wherein: the logicdevice comprises a controller of the mobile structure; the controllingthe mechanical roll stabilizer comprises generating a control signalbased, at least in part, on the determined boresight roll and providingthe control signal to a controller module of the imaging system; and thecontroller module of the imaging system is configured to receive thecontrol signal from the controller of the mobile structure and controlan actuator of the mechanical roll stabilizer, based, at least in part,on the received control signal, to rotate the imaging module tocompensate for the determined boresight roll.
 14. The imaging system ofclaim 11, wherein: the logic device comprises a controller of the mobilestructure; the controlling the mechanical roll stabilizer comprisesproviding the determined boresight roll to a controller module of theimaging system; and the controller module of the imaging system isconfigured to receive the determined boresight roll from the controllerof the mobile structure, generate a control signal based, at least inpart, on the determined boresight roll, and provide the control signalto an actuator of the mechanical roll stabilizer to rotate the imagingmodule to compensate for the determined boresight roll.
 15. The imagingsystem of claim 11, wherein the imaging module comprises a first imagingmodule configured to capture thermal image data according to a first FOVof the first imaging module, the imaging system further comprising: asecond imaging module disposed within the sealed enclosure andconfigured to capture visible spectrum image data along the optical axisof the imaging system according to a second FOV of the second imagingmodule; wherein the mechanical roll stabilizer is coupled to the firstand second imaging modules and configured to provide mechanical rollstabilization of the first and second imaging modules simultaneouslywith respect to the optical axis of the imaging system.
 16. A systemcomprising: an imaging system coupled to a mobile structure, wherein theimaging system comprises an imaging module coupled mechanical rollstabilizer disposed within a sealed enclosure for the imaging system,and wherein the mechanical roll stabilizer is configured to providemechanical roll stabilization of the imaging module with respect to theoptical axis of the imaging system; and a logic device configured tocommunicate with the imaging system and an orientation sensor coupled tothe mobile structure, wherein the logic device is configured to: receiveorientation data from the orientation sensor; determine a boresight rollof the imaging module based, at least in part, on the receivedorientation data; and control the mechanical roll stabilizer tocompensate for the determined boresight roll.
 17. The system of claim16, wherein the logic device comprises a controller module of theimaging system that is disposed within the sealed enclosure or comprisesa controller of the mobile structure, and wherein the controlling themechanical roll stabilizer comprises: generating a control signal based,at least in part, on the determined boresight roll and providing thecontrol signal to an actuator of the mechanical roll stabilizer torotate the imaging module to compensate for the determined boresightroll; or generating a control signal based, at least in part, on thedetermined boresight roll and providing the control signal to acontroller module of the imaging system; and the controller module ofthe imaging system is configured to receive the control signal from thecontroller of the mobile structure and control an actuator of themechanical roll stabilizer, based, at least in part, on the receivedcontrol signal, to rotate the imaging module to compensate for thedetermined boresight roll.
 18. The system of claim 16, wherein theimaging module comprises a first imaging module configured to capturethermal image data according to a first FOV of the first imaging module,the imaging system further comprising: a second imaging module disposedwithin the sealed enclosure and configured to capture visible spectrumimage data along the optical axis of the imaging system according to asecond FOV of the second imaging module; and wherein the mechanical rollstabilizer is coupled to the first and second imaging modules andconfigured to provide mechanical roll stabilization of the first andsecond imaging modules simultaneously with respect to the optical axisof the imaging system.
 19. The system of claim 16, wherein the logicdevice is configured to: aggregate image data provided by the imagingsystem into a passage database; determine a set of potential navigationhazard contacts based, at least in part, on the passage database; anddetermine a set of hazard priorities corresponding to the set ofpotential navigation hazard contacts based, at least in part, on thepassage database.
 20. The system of claim 19, further comprising anoperational state controller coupled to or within the mobile structure,wherein the logic device is configured to: adjust, using the operationalstate controller, the operational state of the mobile structure based,at least in part, on the set of potential navigation hazard contacts,the corresponding set of hazard priorities, and a preselected minimumpriority threshold; wherein the operational state of the mobilestructure comprises a target heading, speed, and/or route for the mobilestructure; and wherein the adjusting the operational state of the mobilestructure comprises adjusting the target heading, speed, and/or routefor the mobile structure to avoid one or more potential navigationhazard contacts with corresponding hazard priorities above thepreselected minimum priority threshold.